Valgrind failures in Apply Launcher's bgworker_quickdie() exit

Hello,

Since libcrypto.so is implicated, Andres asked me off-list if my
changes to random number state initialisation might be linked to
skink's failures beginning 12 or 15 days ago.  It appears not, as it
was green for several runs after that commit.  Looking at the report:

==2802== VALGRINDERROR-BEGIN
==2802== Invalid read of size 8
==2802==    at 0x4DED5A5: check_free (dlerror.c:188)
==2802==    by 0x4DEDAB1: free_key_mem (dlerror.c:221)
==2802==    by 0x4DEDAB1: __dlerror_main_freeres (dlerror.c:239)
==2802==    by 0x55DCF81: __libc_freeres (in /lib/x86_64-linux-gnu/libc-2.28.so)
==2802==    by 0x482D19E: _vgnU_freeres (vg_preloaded.c:77)
==2802==    by 0x478AD3: bgworker_quickdie (bgworker.c:661)
==2802==    by 0x48626AF: ??? (in /lib/x86_64-linux-gnu/libpthread-2.28.so)
==2802==    by 0x556DB76: epoll_wait (epoll_wait.c:30)
==2802==    by 0x4E25B9: WaitEventSetWaitBlock (latch.c:1078)
==2802==    by 0x4E25B9: WaitEventSetWait (latch.c:1030)
==2802==    by 0x4E28C1: WaitLatchOrSocket (latch.c:407)
==2802==    by 0x4E29A6: WaitLatch (latch.c:347)
==2802==    by 0x49E03E: ApplyLauncherMain (launcher.c:1062)
==2802==    by 0x479831: StartBackgroundWorker (bgworker.c:834)
==2802==  Address 0x7fd7e28 is 12 bytes after a block of size 12 alloc'd
==2802==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==2802==    by 0x4C3BD38: CRYPTO_zalloc (in
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==2802==    by 0x4C37F8D: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==2802==    by 0x4C615B9: RAND_DRBG_get0_public (in
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==2802==    by 0x4C615EF: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==2802==    by 0x675D75: pg_strong_random (pg_strong_random.c:135)
==2802==    by 0x4848EB: RandomCancelKey (postmaster.c:5251)
==2802==    by 0x484909: assign_backendlist_entry (postmaster.c:5822)
==2802==    by 0x4873BA: do_start_bgworker (postmaster.c:5692)
==2802==    by 0x487701: maybe_start_bgworkers (postmaster.c:5955)
==2802==    by 0x4878C2: reaper (postmaster.c:2940)
==2802==    by 0x48626AF: ??? (in /lib/x86_64-linux-gnu/libpthread-2.28.so)
==2802==
==2802== VALGRINDERROR-END

The function __libc_freeres is a special glibc entry point provided
for leak checkers to call explicitly if they want glibc to clean up
after itself (normally it doesn't bother).  The specific thing being
cleaned up here is a piece of thread local storage that belongs to the
dynamic linker support code:

https://github.com/lattera/glibc/blob/master/dlfcn/dlerror.c#L228

Since we don't see strcmp() or free() at the top of the stack (and
assuming they aren't inlined), I think the line numbers must line up
with current glibc HEAD as of today, and it must be failing on
accessing rec->errstring at line 188, meaning that rec (the value
stored as a thread specific key) is a bad pointer.  That's quite
strange and I don't have an explanation; if libcrypto overran its
buffer, for example, that would perhaps trash rec->errstring but we'd
still be able to read the pointer itself.  So I wonder if libcrypto.so
is a red herring here.

It's Debian unstable, which could be a factor.  Bugs in glibc?
That's 2.28, out for 3 months now, but then why only in Apply
Launcher? Did we trash 'key', or the thread specific pointer table, or
is my assessment above wrong, and somehow it's really errstring that
is a bad pointer (which would allow for a more mundane explanation,
like someone trashed a bit of heap memory by overrunning a buffer)?

-- 
Thomas Munro
http://www.enterprisedb.com