RE: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

It's a new project that need security compliance , SSL is a MUST here , and pgaudit,set_user is installed here too to
meetingthe compliance request.  We test renew SSL certificate, and change the ssl_cert_file and ssl_key_file parameter
torenewed ssl certificates.  
ssl = on
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'

ssl_crl_file = ''
#ssl_min_protocol_version = 'TLSv1.2'
ssl_ca_file = '/var/lib/pgsql/sslrenew/idtrca.cer'
#ssl_cert_file = '/var/lib/pgsql/sslrenew/postgres-109798.crt'
#ssl_key_file = '/var/lib/pgsql/sslrenew/postgres-109798.key'

ssl_cert_file = '/var/lib/pgsql/sslrenew/postgres014-110388.crt'
ssl_key_file = '/var/lib/pgsql/sslrenew/postgres014-11038.key'

--
shared_preload_libraries = 'orafce,pgaudit,pg_cron,pg_stat_statements,pg_prewarm,set_user'
pgaudit.log_catalog='on'
pgaudit.log_level='log'
pgaudit.log_parameter=on
pgaudit.log_statement_once=off
pgaudit.log='all, -misc'
pgaudit.log='ddl,role'
pgaudit.role='postgres,jamet'

#set_user
set_user.superuser_whitelist = '+dba'
#set_user.superuser_allowlist = '+dba'
set_user.block_log_statement=on
#set_user.nosuperuser_target_whitelist = ''
set_user.nosuperuser_target_allowlist = ''

#pre_warm
pg_prewarm.autoprewarm = true
pg_prewarm.autoprewarm_interval = 600


the Operating system got some security hardening too, too meet compliance requirement.   The OpenSSL 1.1.1g with FIPS
enabled. 
$ openssl version
OpenSSL 1.1.1g FIPS  21 Apr 2020


Yes, interesting thing is when I remove all extensions and try the test again, then install orafce, pg_background,
pgaudit,looks like not reproduced the issue, until install set_user rpm it's ok, but when create extension again,
reproducedthe issue.  

=# \dx
                                                       List of installed extensions
        Name        | Version |   Schema   |                                          Description
--------------------+---------+------------+----------------------------
--------------------+---------+------------+----------------------------
--------------------+---------+------------+----------------------------
--------------------+---------+------------+-----------
 amcheck            | 1.2     | public     | functions for verifying relation integrity
 orafce             | 3.15    | public     | Functions and operators that emulate a subset of functions and packages
fromthe Oracle RDBMS 
 pageinspect        | 1.8     | public     | inspect the contents of database pages at a low level
 pg_background      | 1.0     | public     | Run SQL queries in the background
 pg_buffercache     | 1.3     | public     | examine the shared buffer cache
 pg_cron            | 1.4     | public     | Job scheduler for PostgreSQL
 pg_freespacemap    | 1.2     | public     | examine the free space map (FSM)
 pg_permissions     | 1.1     | public     | view object permissions and compare them with the desired state
 pg_stat_statements | 1.8     | public     | track planning and execution statistics of all SQL statements executed
 pgaudit            | 1.5     | public     | provides auditing functionality
 pgstattuple        | 1.5     | public     | show tuple-level statistics
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
 postgres_fdw       | 1.0     | public     | foreign-data wrapper for remote PostgreSQL servers
 set_user           | 3.0     | public     | similar to SET ROLE but with added logging
(14 rows)


Thanks,

James

-----Original Message-----
From: Dmitry Dolgov <9erthalion6@gmail.com>
Sent: Tuesday, December 14, 2021 11:46 PM
To: Michael Paquier <michael@paquier.xyz>
Cc: James Pang (chaolpan) <chaolpan@cisco.com>; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

> On Mon, Dec 13, 2021 at 08:10:57PM +0900, Michael Paquier wrote:
> On Mon, Dec 13, 2021 at 07:06:16AM +0000, James Pang (chaolpan) wrote:
> > Edit postgresql.conf to change ssl_certificate parameter ,
>
> Do you mean ssl_cert_file here?  Also, something that's not completely
> clear to me is if this is a problem with a vanilla PostgreSQL instance
> or if this is related to the pgaudit extension set_user, as it has
> been mentioned as one potential origin of the problem upthread, but
> you are not telling if this is the case here.  So what do you have for
> shared_preload_libraries in this crash?
>
> > #9  0x00007ff49a78059c in ssl_cert_clear_certs () from
> > /lib64/libssl.so.1.1
> > #10 0x00007ff49a780645 in ssl_cert_free () from /lib64/libssl.so.1.1
> > #11 0x00007ff49a78a25c in SSL_CTX_free () from /lib64/libssl.so.1.1
> > #12 0x000000000068b6b8 in be_tls_init ()
> > #13 0x00000000007271e1 in SIGHUP_handler ()
>
> Why is secure_initialize() not showing up in this stack?  That would
> be the caller of be_tls_init() in the SIGHUP handler.  The version of
> OpenSSL you are linking your binaries to would be useful here.  That
> would be a 1.1.0 or a 1.1.1, no?  Any specific minor version letter?

I think I can actually reproduce the issue. In my case the stack is fine, it contains secure_initialize, and overall it
lookslike some sort of memory corruption -- at least openssl gets segfault because it can't access some memory address
ittries to verify in asn1_primitive_free. 
Not sure yet why, investigating.