Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

On 1/12/24 14:03, David Ventimiglia wrote:

Reply to list also.
Ccing list
> Hi Adrian,
> 
> I left out the creation of the slot both from the SO question and from 
> this mailing list question, because I believe it's a red herring.  I run 
> into the same problem with the default output plugin as I do with the 
> wal2json plugin.  However, the problem is a little different from what I 
> described.  It turns out it's not output redirection from pg_recvlogical 
> sending to stdout that fails.  Rather, it's output redirection from 
> pg_recvlogical sending to stdout and then passed through a pipeline that 
> fails.  Or something like that.
> 
> This works.  The sample.txt file is non-empty.
> 
>     pg_recvlogical -d postgres --slot=test --create-slot
>     pg_recvlogical -d postgres -n --slot=test --start -f - > sample.txt
> 
> 
> This does not work.  The sample.txt file is empty.
> 
>     pg_recvlogical -d postgres --slot=test --create-slot
>     pg_recvlogical -d postgres -n --slot=test --start -f - | awk
>     '{print}' > sample.txt
> 
> 
> Weirdly, this works.  The sample.txt file is non-empty.
> 
>     pg_recvlogical -d postgres --slot=test --create-slot
>     pg_recvlogical -d postgres -n --slot=test --start -f - | cat >
>     sample.txt
> 
> 
> FWIW, this is demonstrated in this screen-cast 
> <https://asciinema.org/a/631166>.  I suspect it has something to do with 
> the flushing of buffers, as other people have suggested.
> 
> Thanks!
> David
> 
> 
> 
> 
> On Fri, Jan 12, 2024 at 1:48 PM Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> wrote:
> 
>     On 1/12/24 11:34, David Ventimiglia wrote:
>      > Hello! How do I redirect logical decoding output from the
>     PostgreSQL CLI
>      > tool |pg_recvlogical| either to a file or to another command via
>     a pipe?
>      > I ask because when I try the obvious, no output is recorded or sent:
>      >
>      > |pg_recvlogical -d postgres --slot test --start -f - >>
>     sample.jsonl |
>      >
>      > Lest there be any confusion, I already created the slot in an
>     earlier
>      > step. Moreover, I can verify that if I omit the output
>     redirection |>>
>      > sample| then it does work, insofar as it emits the expected change
>      > events when I perform DML in another terminal window. When I
>     include the
>      > redirection (or alternatively, set up a pipeline), then nothing
>     happens.
> 
>     You left out the important part of the SO question:
> 
>     pg_recvlogical -d postgres --slot test --create-slot -P wal2json
> 
>     I can get:
> 
>     pg_recvlogical -d postgres --slot test --start -f - >> sample.jsonl
> 
>     to work when I do:
> 
>     pg_recvlogical -d postgres --slot test --create-slot
> 
>     wal2json seems to be the issue.
> 
>      >
>      > Note that I am aware of the option to pass a filename to the -f
>     switch
>      > to write to a file.  That works, but it's not what I'm after
>     because it
>      > doesn't help update my mental model of how this is supposed to work.
>      > Based on my current (flawed) mental model built up from command line
>      > experience with other tools, this /should/ work.  I should be
>     able to
>      > send the output to stdout and then redirect it to a file.  It
>     surprises
>      > me that I cannot.
>      >
>      > Anyway, thanks!
>      >
>      > Best,
>      >
>      > David
>      >
> 
>     -- 
>     Adrian Klaver
>     adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com