RE: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters - Mailing list pgsql-bugs

try to install debug_info and get stack,
1. use coredump ,
 ]$ gdb -q -c /pgdata/core.1317550.sig11.1639122870s /usr/pgsql-13/bin/postgres
Reading symbols from /usr/pgsql-13/bin/postgres...Reading symbols from .gnu_debugdata for
/usr/pgsql-13/bin/postgres...(nodebugging symbols found)...done. 
(no debugging symbols found)...done.

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 1317550]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/pgsql-13/bin/postgres'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f72e3290094 in asn1_string_embed_free () from /lib64/libcrypto.so.1.1

2. when gdb log ,
Program received signal SIGHUP, Hangup.
0x00007f4fb438e25b in select () from /lib64/libc.so.6
Continuing.

Program received signal SIGHUP, Hangup.
0x00007f4fb438e25b in select () from /lib64/libc.so.6
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f4fb5eef094 in asn1_string_embed_free () from /lib64/libcrypto.so.1.1
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

Should I install debug info for set_user module too?

Thanks,

James

-----Original Message-----
From: James Pang (chaolpan)
Sent: Thursday, December 9, 2021 11:34 AM
To: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: RE: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

    Looks like this issue is related with "set_user" extension, I removed all extensions , pg_reload_conf() works
withoutissue.  When I installed and enable "set_user" extension, the issue got reproduced. 
    shared_preload_libraries = 'orafce,pgaudit,pg_cron,pg_stat_statements,pg_prewarm,set_user'
#set_user
set_user.superuser_whitelist = '+dba'
#set_user.superuser_allowlist = '+dba'
set_user.block_log_statement=on
set_user.nosuperuser_target_whitelist = ''
#set_user.nosuperuser_target_allowlist = ''

   Will try to get and update the stack.

James

-----Original Message-----
From: Dmitry Dolgov <9erthalion6@gmail.com>
Sent: Wednesday, December 8, 2021 9:46 PM
To: James Pang (chaolpan) <chaolpan@cisco.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

> The following bug has been logged on the website:
>
> Bug reference:      17326
> Logged by:          James Pang
> Email address:      chaolpan@cisco.com
> PostgreSQL version: 13.4
> Operating system:   RHEL8.4
> Description:
>
> we need SSL enabled for our production env, when I test renew a ssl
> certificate , and reload_conf, it crashed. even with same certificate
> and ssl parameters, run reload_conf often lead to Postgres crash. For
> example
> :
>
>  =# select name,setting from pg_settings where name like 'ssl_%' order by name;
>                   name                  |                setting
> ----------------------------------------+-----------------------------
> ----------------------------------------+--
> ----------------------------------------+--------
>  ssl_ca_file                            |
> /var/lib/pgsql/sslcerts/awstestca.crt
>  ssl_cert_file                          |
> /var/lib/pgsql/sslcerts/server.crt
>  ssl_ciphers                            | HIGH:MEDIUM:+3DES:!aNULL
>  ssl_crl_file                           |
>  ssl_dh_params_file                     |
>  ssl_ecdh_curve                         | prime256v1
>  ssl_key_file                           |
> /var/lib/pgsql/sslcerts/server.key
>  ssl_library                            | OpenSSL
>  ssl_max_protocol_version               |
>  ssl_min_protocol_version               | TLSv1.2
>  ssl_passphrase_command                 |
>  ssl_passphrase_command_supports_reload | off
>  ssl_prefer_server_ciphers              | on
> (13 rows)
>
>  =# select pg_reload_conf();
>  pg_reload_conf
> ----------------
>  t
> (1 row)
>
>  =# select pg_reload_conf();
>  pg_reload_conf
> ----------------
>  t
> (1 row)
>
>  =# select pg_reload_conf();
> FATAL:  terminating connection due to unexpected postmaster exit server closed the connection unexpectedly
>         This probably means the server terminated abnormally
>         before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
>
> On Wed, Dec 08, 2021 at 06:22:11AM +0000, James Pang (chaolpan) wrote:
> From postgres logs , it show
> 2021-12-08 03:57:55.826 UTC::@:[1291058]:[9-1]:2021-12-08 03:33:21
> UTC:LOG:  received SIGHUP, reloading configuration files
> 2021-12-08 03:58:02.832 UTC::@:[1291058]:[10-1]:2021-12-08 03:33:21
> UTC:LOG:  received SIGHUP, reloading configuration files
> 2021-12-08 03:58:03.143
> UTC:10.240.212.242(58646):jamet@jamet:[1291076]:[9-1]:2021-12-08
> 03:33:24 UTC:testsubLOG:  disconnection: session time: 0:24:38.967
> user=jamet database=jamet host=10.240.212.242 port=58646
> 2021-12-08 03:58:03.147
> UTC:[local]:postgres@jamet:[1291397]:[3-1]:2021-12-08 03:57:02
> UTC:psqlFATAL:  terminating connection due to unexpected postmaster
> exit
> 2021-12-08 03:58:03.147
> UTC:[local]:postgres@jamet:[1291397]:[4-1]:2021-12-08 03:57:02
> UTC:psqlLOG:  disconnection: session time: 0:01:00.405 user=postgres
> database=jamet host=[local]

Hi,

Thanks for reporting the issue. Any chance to get a stack trace corresponding to the crash, e.g. like in [1]?

[1]: https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD