Re: Non-blocking synchronization in libpq using pipeline mode

On Tue, 19 Mar 2024 12:49:23 +0100
Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

> On 2024-Mar-06, PG Doc comments form wrote:
>
> > The following documentation comment has been logged on the website:
> >
> > Page: https://www.postgresql.org/docs/16/libpq-pipeline-mode.html
> > Description:
> >
> > The calls to PQpipelineSync and PQsendFlushRequest may either report failure
> > or success, but not that data could not be written as of yet because the
> > request would block.
>
> Is this a literal quote?  If so, where do you see it?

It is not a literal quote. It is a deduction from the function
signatures and the documented return values:

int PQpipelineSync(PGconn *conn);
int PQsendFlushRequest(PGconn *conn);

which, according to the documentation, only ever return 0 or 1,
indicating success or failure. (Compare with PQflush, which can return
three different values instead of two.)

>
> > Does this mean that
> >
> > 1. these functions will always block when invoked and the socket is not
> > ready to accept the number of bytes that need to be written (assuming the
> > number of bytes required to be written is greater than one byte)
>
> No.

That seems true for "PQsendFlushRequest" as the documentation specifically
states that:

"Note that the request is not itself flushed to the server
automatically; use PQflush if necessary."

However, that sentence only applies to "PQsendFlushRequest", not to
"PQpipelineSync". On the contrary, section "34.5.1.1. Issuing Queries"
reads:

"[Flushing to the server] occurs when PQpipelineSync is used to
establish a synchronization point in the pipeline, or when PQflush is
called."

As I understand it, this means that PQpipelineSync will flush the
client-side's output, while PQsendFlushRequest won't flush the
client-side. So both commands act differently with regard to flushing,
is that right?

If that is the case, then the question is what happens if you call
"PQpipelineSync" in non-blocking mode. That isn't clear to me (and I
believe it is not explicitly made clear in the documentation).

In section "34.4. Asynchronous Command Processing", we find:

"In the nonblocking state, successful calls to PQsendQuery, PQputline,
PQputnbytes, PQputCopyData, and PQendcopy will not block; their changes
are stored in the local output buffer until they are flushed.
Unsuccessful calls will return an error and must be retried."

There is no reference to "PQpipelineSync" there. Does that mean that
"PQsetnonblocking" has no effect on "PQpipelineSync"'s blocking
behavior? If so, is it correct that "PQpipelineSync" always blocks when
the socket is not ready for writing (since, as stated in the
documentation, it does flush)?

I think section 34.5.1.4 should clarifiy how exactly PQpipelineSync
acts in regard to flushing and blocking in that matter. It doesn't seem
very clear at the moment.

>
> > 2. the synchronization or flush request need to be flushed manually with
> > successive PQflush calls
>
> Yes.

Regarding "PQpipelineSync", the documentation implies otherwise. See
cite above:

"[Flushing to the server] occurs when PQpipelineSync is used to
establish a synchronization point in the pipeline, or when PQflush is
called."

That sentence doesn't make sense if PQpipelineSync wouldn't flush.

>
> > or
> >
> > 3. the functions will return an error condition when the connection is
> > non-blocking and no data could be written.
>
> No.
>
> > Any clarification in the documentation would be appreciated.
>
> So I checked https://www.postgresql.org/docs/16/libpq-pipeline-mode.html
> and it has this, under PQsendFlushRequest:
>
>    The server flushes its output buffer automatically as a result of
>    PQpipelineSync being called, or on any request when not in pipeline
>    mode; this function is useful to cause the server to flush its output
>    buffer in pipeline mode without establishing a synchronization point.
>    Note that the request is not itself flushed to the server
>    automatically; use PQflush if necessary.
>
> which I think answers what you are asking.

It answers the behavior for "PQsendFlushRequest", but it's still
unclear how "PQpipelineSync" will act.

>
> --
> Álvaro Herrera        Breisgau, Deutschland  ?  https://www.EnterpriseDB.com/

Kind Regards,
Jan Behrens