Обсуждение: [BUG] PostgreSQL crashes with ThreadSanitizer during early initialization

Hi hackers, I've found a bug that causes PostgreSQL to crash during startup when built with ThreadSanitizer
(-fsanitize=thread).

My environment
Ubuntu 24.04.1 LTS (kernel 6.14.0-29-generic)
clang 18
PostgreSQL 17.2
Build Configuration: ./configure --enable-debug --enable-cassert CFLAGS="-fsanitize=thread -g"

PostgreSQL compiled with ThreadSanitizer (-fsanitize=thread) crashes with SIGSEGV during program initialization, before
reachingmain(). 

Steps to Reproduce

1. Configure PostgreSQL with ThreadSanitizer
2.  ./configure --enable-debug CFLAGS="-fsanitize=thread -g"
3. make
4. Run any PostgreSQL command:  ./postgres --version

Expected Behavior: Program should start normally and display version information.
Actual Behavior: Segmentation fault during early initialization

Root Cause: The __ubsan_default_options() function in main.c is compiled with TSan instrumentation, creating a circular
dependencyduring sanitizer runtime initialization. 
1. TSan initialization calls __ubsan_default_options()
2. TSan tries to instrument the function
3. Instrumentation requires initialized ThreadState
4. ThreadState isn't ready because TSan init isn't complete
5. Segfault/crash occurs

Proposed Fix: Move __ubsan_default_options() to a separate compilation unit built without sanitizer instrumentation.
The below attached patch moves the function to a separate compilation unit with a custom Makefile rule that uses
-fno-sanitize=thread,address,undefined.The reached_main check is preserved to avoid calling getenv() before libc is
fullyinitialized and to handle cases where set_ps_display() breaks /proc/$pid/environ. 

Please let me know if you have any questions or would like further details.
Thanks & Regards,
Emmanuel Sibi

Hi Emmanuel,

> Hi hackers, I've found a bug that causes PostgreSQL to crash during startup when built with ThreadSanitizer
(-fsanitize=thread).
>
> [...]

Thanks for reporting this. Did you investigate whether Meson also has
this issue? Fixing anything for Autotools arguably has low priority
since we are going to get rid of it in the near future, but Meson is
another matter.

-- 
Best regards,
Aleksander Alekseev

Hi Aleksander,
> Thanks for reporting this. Did you investigate whether Meson also has
> this issue? Fixing anything for Autotools arguably has low priority
> since we are going to get rid of it in the near future, but Meson is
> another matter.

Thanks for the reply. Yes, I tested with Meson and confirmed the same
issue occurs.
When building PostgreSQL 17.2 with ThreadSanitizer:
meson setup builddir \
   --prefix=/path/to/install \
   --buildtype=debug \
   -Dcassert=true \
   -Dtap_tests=enabled \
   -Db_lto=false \
   -Db_sanitize=thread \
   -Db_lundef=false \
   -Dc_args="-O0 -g -gdwarf-2 -fno-omit-frame-pointer"

The postgres binary segfaults during early initialization, exactly as
with the Autotools build.
Applying the patch I submitted, which moves __ubsan_default_options()
to a separate compilation unit built without sanitizer
instrumentation, successfully resolves the segfault.

Thanks & regards,
Emmanuel Sibi

Hi Emmanuel,

> Thanks for the reply. Yes, I tested with Meson and confirmed the same
> issue occurs.
> [...]

OK, thanks for the details. Please don't forget to register your patch
on the nearest open commitfest:

https://commitfest.postgresql.org/56/

Otherwise it can be lost.

-- 
Best regards,
Aleksander Alekseev

On 9/9/25 10:37 PM, Aleksander Alekseev wrote:
> Hi Emmanuel,
> 
>> Thanks for the reply. Yes, I tested with Meson and confirmed the same
>> issue occurs.
>> [...]
> 
> OK, thanks for the details. Please don't forget to register your patch
> on the nearest open commitfest:
> 
> https://commitfest.postgresql.org/56/
> 
> Otherwise it can be lost.
> 
I tested this patch. postgres -V no longer crashes.

There is a minor issue. I'm not sure if it's caused by this patch.

The database can only be shut down using the immediate mode.

2025-09-10 08:29:52.667 CST [53336] LOG:  received immediate shutdown 
request
2025-09-10 08:29:57.771 CST [53336] LOG:  issuing SIGKILL to 
recalcitrant children
2025-09-10 08:29:57.897 CST [53336] LOG:  database system is shut down


Other modes can only indicate that a shutdown request has been received 
but cannot actually stop it.

2025-09-10 08:21:13.445 CST [53280] LOG:  received fast shutdown request
2025-09-10 08:21:13.446 CST [53280] LOG:  aborting any active transactions
2025-09-10 08:21:13.587 CST [53280] LOG:  background worker "logical 
replication launcher" (PID 53290) exited with exit code 1
2025-09-10 08:21:13.588 CST [53284] LOG:  shutting down
2025-09-10 08:21:13.588 CST [53284] LOG:  checkpoint starting: shutdown fast
2025-09-10 08:21:13.607 CST [53284] LOG:  checkpoint complete: wrote 0 
buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 
recycled; write=0.017 s, sync=0.001 s, total=0.020 s; sync files=2, 
longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; 
lsn=0/0178A340, redo lsn=0/0178A340

The smart mode is the same as well.


Best regards,
Quan Zongliang

Hi Quan, Thanks for testing the patch! I'm glad it resolves the startup crash.
> I tested this patch. postgres -V no longer crashes.

Regarding the shutdown issue - I tested extensively with
ThreadSanitizer enabled using both build configurations I mentioned
earlier, and all shutdown modes work correctly in my environment:
> There is a minor issue. I'm not sure if it's caused by this patch.
> The database can only be shut down using the immediate mode.

Fast shutdown:
2025-09-15 20:20:07.454 IST [28229] LOG: received fast shutdown request
[...]
2025-09-15 20:20:07.574 IST [28229] LOG: database system is shut down
Smart shutdown:
2025-09-15 20:28:14.271 IST [31263] LOG: received smart shutdown request
[...]
2025-09-15 20:28:14.399 IST [31263] LOG: database system is shut down

All modes complete within seconds with the final "database system is
shut down" message, unlike the hang you're experiencing.
The patch only moves __ubsan_default_options() to a separate
compilation unit to avoid TSan initialization issues during startup.
It doesn't modify any shutdown logic or signal handling code.
My Environment: Ubuntu 24.04.1, clang 18, PostgreSQL 17.2.

Best regards,
Emmanuel

On 9/15/25 11:19 PM, Emmanuel Sibi wrote:
> Hi Quan, Thanks for testing the patch! I'm glad it resolves the startup crash.
> 
> Fast shutdown:
> 2025-09-15 20:20:07.454 IST [28229] LOG: received fast shutdown request
> [...]
> 2025-09-15 20:20:07.574 IST [28229] LOG: database system is shut down
> Smart shutdown:
> 2025-09-15 20:28:14.271 IST [31263] LOG: received smart shutdown request
> [...]
> 2025-09-15 20:28:14.399 IST [31263] LOG: database system is shut down
> 
> All modes complete within seconds with the final "database system is
> shut down" message, unlike the hang you're experiencing.
> The patch only moves __ubsan_default_options() to a separate
> compilation unit to avoid TSan initialization issues during startup.
> It doesn't modify any shutdown logic or signal handling code.
> My Environment: Ubuntu 24.04.1, clang 18, PostgreSQL 17.2.
> 
Great! My OS is macOS 15.6.1. I will continue to test to confirm if 
there are any other issues. If so, I will create a separate patch.

Emmanuel Sibi <emmanuelsibi.mec@gmail.com> writes:
> Root Cause: The __ubsan_default_options() function in main.c is compiled with TSan instrumentation, creating a
circulardependency during sanitizer runtime initialization. 
> 1. TSan initialization calls __ubsan_default_options()
> 2. TSan tries to instrument the function
> 3. Instrumentation requires initialized ThreadState
> 4. ThreadState isn't ready because TSan init isn't complete
> 5. Segfault/crash occurs

Hmm.  I wonder what is the argument that this is not a bug
of UBSan itself, as it seems to make __ubsan_default_options()
next door to impossible to use safely.

> Proposed Fix: Move __ubsan_default_options() to a separate compilation unit built without sanitizer instrumentation.

I do not love this fix, as it requires exposing reached_main globally,
not to mention getting both of our build systems involved in the hack.
Another problem is that it only defends against a limited set of
sanitizers, though presumably every single one is broken in the same
way (compare [1]).

I tried this as an alternative solution, but it didn't seem to help:

diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index bdcb5e4f261..cc63da97360 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -500,6 +500,12 @@ check_root(const char *progname)
  * seem necessary to only compile it conditionally.
  */
 const char *__ubsan_default_options(void);
+
+#if __has_attribute(no_sanitize)
+__attribute__((no_sanitize("thread")))
+__attribute__((no_sanitize("address")))
+__attribute__((no_sanitize("undefined")))
+#endif
 const char *
 __ubsan_default_options(void)
 {

This is of course no better on the "limited set of sanitizers"
angle, but it at least keeps the hack localized.  Of course,
if it doesn't work that's all moot, but I wonder why not ---
seems like it should have about the same effect as your proposal.
(I did verify that clang complains if I misspell a no_sanitize
argument, so it's not that the syntax has no effect at all.)

Anyway, I think really a bug report to the UBSan folk asking
how one is supposed to use __ubsan_default_options() safely
might be productive.

            regards, tom lane

[1] https://www.postgresql.org/message-id/dbf77bf7-6e54-ed8a-c4ae-d196eeb664ce%40gmail.com

On Tue, Nov 4, 2025 at 2:39 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Another problem is that it only defends against a limited set of
> sanitizers, though presumably every single one is broken in the same
> way (compare [1]).

How about __attribute__((disable_sanitizer_instrumentation)) ? LLVM's
own tests make some use of this [1].

--Jacob

[1] https://github.com/llvm/llvm-project/blob/2b4ac6629/compiler-rt/test/sanitizer_common/TestCases/dlsym_alloc.c

Jacob Champion <jacob.champion@enterprisedb.com> writes:
> On Tue, Nov 4, 2025 at 2:39 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Another problem is that it only defends against a limited set of
>> sanitizers, though presumably every single one is broken in the same
>> way (compare [1]).

> How about __attribute__((disable_sanitizer_instrumentation)) ? LLVM's
> own tests make some use of this [1].

Hah, thanks for the research!  For me, this stops the failure
(on RHEL9 with clang version 19.1.7):

diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index bdcb5e4f261..1bd63ec9184 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -500,6 +500,10 @@ check_root(const char *progname)
  * seem necessary to only compile it conditionally.
  */
 const char *__ubsan_default_options(void);
+
+#if __has_attribute(disable_sanitizer_instrumentation)
+__attribute__((disable_sanitizer_instrumentation))
+#endif
 const char *
 __ubsan_default_options(void)
 {

Assuming that works for Emmanuel, we could wrap it in a
pg_disable_sanitizer_instrumentation macro, or just use it
as-is.  I don't have a strong preference --- any thoughts?

(It could do with a comment, either way.)

            regards, tom lane

On Tue, Nov 4, 2025 at 5:19 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Hah, thanks for the research!  For me, this stops the failure
> (on RHEL9 with clang version 19.1.7):

Awesome!

> Assuming that works for Emmanuel, we could wrap it in a
> pg_disable_sanitizer_instrumentation macro, or just use it
> as-is.  I don't have a strong preference --- any thoughts?

No preference on my end. (If a second place to use it pops up, we
could wrap it then.)

--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:
> On Tue, Nov 4, 2025 at 5:19 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Assuming that works for Emmanuel, we could wrap it in a
>> pg_disable_sanitizer_instrumentation macro, or just use it
>> as-is.  I don't have a strong preference --- any thoughts?

> No preference on my end. (If a second place to use it pops up, we
> could wrap it then.)

Yeah, that's what I concluded after sleeping on it.  Right now
it seems unlikely that there will be more usages, so adding a
macro in c.h would just slow down the build (admittedly only
microscopically) for no gain.  If we find additional usages
then we can revisit that tradeoff.

            regards, tom lane