Обсуждение: Non-blocking archiver process

Hello,

We've noticed a behavior that seems surprising to us. 
Since DROP DATABASE now waits for a ProcSignalBarrier, it can hang up 
indefinitely if the archive_command hangs. 

The reason for this is that the builtin archive module doesn't process any 
interrupts while the archiving command is running, as it's run with a system() 
call, blocking undefintely. 

Before rushing on to implement a non-blocking archive library (perhaps using 
popen or posix_spawn, while keeping other systems in mind), what unintended 
consequences would it have to actually run the archive_command in a non-
blocking way, and checking interrupts while it runs ?

Thanks !

--
Ronan Dunklau

On Fri, Jul 04, 2025 at 08:46:08AM +0200, Ronan Dunklau wrote:
> We've noticed a behavior that seems surprising to us. 
> Since DROP DATABASE now waits for a ProcSignalBarrier, it can hang up 
> indefinitely if the archive_command hangs. 
> 
> The reason for this is that the builtin archive module doesn't process any 
> interrupts while the archiving command is running, as it's run with a system() 
> call, blocking undefintely. 
> 
> Before rushing on to implement a non-blocking archive library (perhaps using 
> popen or posix_spawn, while keeping other systems in mind), what unintended 
> consequences would it have to actually run the archive_command in a non-
> blocking way, and checking interrupts while it runs ?

I can't think of any unintended consequences.  I think we just missed this
when adding the first use of ProcSignalBarrier (v15).  Making this easier to
miss, archiver spent most of its history not connecting to shared memory.  Its
shared memory connection appeared in v14.

On 05.07.25 05:01, Noah Misch wrote:
> On Fri, Jul 04, 2025 at 08:46:08AM +0200, Ronan Dunklau wrote:
>> We've noticed a behavior that seems surprising to us.
>> Since DROP DATABASE now waits for a ProcSignalBarrier, it can hang up
>> indefinitely if the archive_command hangs.
>>
>> The reason for this is that the builtin archive module doesn't process any
>> interrupts while the archiving command is running, as it's run with a system()
>> call, blocking undefintely.
>>
>> Before rushing on to implement a non-blocking archive library (perhaps using
>> popen or posix_spawn, while keeping other systems in mind), what unintended
>> consequences would it have to actually run the archive_command in a non-
>> blocking way, and checking interrupts while it runs ?
> 
> I can't think of any unintended consequences.  I think we just missed this
> when adding the first use of ProcSignalBarrier (v15).  Making this easier to
> miss, archiver spent most of its history not connecting to shared memory.  Its
> shared memory connection appeared in v14.

I've taken some time, mostly for WIN32, to implement an interruptible 
version of archive_command. My WIN32 days are long behind me, so it's 
quite possible that this has some faults I'm not seeing. Then again, it 
passes CI.
I failed to make it work in WIN32 with popen since the handles it 
returns can't be made non-blocking so this change is a bit bigger.

@Ronan: Let me now if you'd like to be attributed more, I took some 
inspiration from a private repos with your prototype.

I don't know if I should add that to the running commitfest for PG19 or 
if this is something that would need to be backported. Just let me know.

Thanks,
Patrick

> On Nov 10, 2025, at 18:41, BharatDB <bharatdbpg@gmail.com> wrote:
>
> Dear PostgreSQL Community,
> I found 8 Bugs in `shell_archive.c`
>
> While studying how `archive_command` works, I discovered **8 real issues** that affect reliability, performance, and
safety: 
> | # | Bug | Impact | |---|-----|--------| | 1 | Infinite loop on command failure | Archiver hangs forever | | 2 |
Unreachablecode after `return` | Dead logic | | 3 | Discarded `stdout` from archive command | `DROP DATABASE` hangs for
fullcommand duration | | 4 | Aggressive polling with `Sleep(100)` | CPU waste (already fixed in core) | | 5 |
`malloc`/`free`in backend | **Memory corruption risk** | | 6 | Poor variable names (`dwRc`, `bytesRead`) | Hard to read
|| 7 | Manual `popen` / `CreateProcess` | Missed PG infrastructure | | 8 | Redundant `return;` in `void` function |
Styleissue |  
> I refactored `src/backend/archive/shell_archive.c` with **PostgreSQL-style fixes**:
> - Replaced `popen()` and `CreateProcess()` → `OpenPipeStream()` - Used `fileno(archiveFile)` → `archiveFd` correctly
-Switched to `palloc()` / `pfree()` for memory safety - Renamed variables: `dwRc` → `exit_code`, `bytesRead` → `nread`,
etc.- Read `stdout` to prevent `DROP DATABASE` hangs - Used `WaitLatchOrSocket()` for interruptible waiting - Removed
redundant`return;` and dead code This is my contribution to improve the PostgreSQL archiver process. I have attached a
patchimplementing the discussed fixes — please review and share any suggestions or feedback. Thanks ! 
> Lakshmi
>

Thanks for the patch, but I just cannot pass build in my end:
```
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla
-Werror=unguarded-availability-new-Wendif-labels -Wmissing-format-attribute -Wcast-function-type -Wformat-security
-Wmissing-variable-declarations-fno-strict-aliasing -fwrapv -fexcess-precision=standard
-Wno-unused-command-line-argument-Wno-compound-token-split-by-macro -Wno-format-truncation
-Wno-cast-function-type-strict-g -O2 -I../../../src/include -isysroot
/Library/Developer/CommandLineTools/SDKs/MacOSX15.5.sdk  -I/opt/homebrew/Cellar/icu4c@77/77.1/include    -c -o
shell_archive.oshell_archive.c -MMD -MP -MF .deps/shell_archive.Po 
shell_archive.c:19:10: fatal error: 'latch.h' file not found
   19 | #include "latch.h"  /* For WaitLatchOrSocket */
      |          ^~~~~~~~~
1 error generated.
make[3]: *** [shell_archive.o] Error 1
make[2]: *** [archive-recursive] Error 2
make[1]: *** [all-backend-recurse] Error 2
make: *** [all-src-recurse] Error 2
```

Also, as a general comment, you need do pgident on the file you changed.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/