Обсуждение: Client-server communication for FETCH
Hello community,
I have a simple C program using libpq V18 to FETCH one row at a time (see below). Does the server send multiple rows to the client where they are cached for satisfying the FETCH statements, or does it just send one row at a time since that's all that each FETCH statement is asking for? If it's sending multiple rows at a time, is there some way that I can observe this e.g. some kind of trace info that would show it? Or where in the libpq source would I look to see how the client is retrieving rows from the server?
I have done extensive searching to try and find the definitive answer to this. The searches indicate that libpq supports the concept of a client-side cursor, where it has a cache of rows sent by the server and uses that cache to perform each FETCH, but I'd like to be able to verify whether this is true or not.
Thanks,
Tim
res = PQexec(conn, "DECLARE cur CURSOR FOR select * from table1");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
printf("DECLARE CURSOR failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
PQclear(res);
res = PQexec(conn, "FETCH in cur");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("FETCH failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
PQclear(res);
res = PQexec(conn, "FETCH in cur");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("FETCH failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
printf("DECLARE CURSOR failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
PQclear(res);
res = PQexec(conn, "FETCH in cur");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("FETCH failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
PQclear(res);
res = PQexec(conn, "FETCH in cur");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("FETCH failed: %s", PQerrorMessage(conn));
PQclear(res);
exit(1);
}
Tim Fors <tim4stheenchanter@gmail.com> writes:
> I have a simple C program using libpq V18 to FETCH one row at a time (see
> below). Does the server send multiple rows to the client where they are
> cached for satisfying the FETCH statements, or does it just send one row at
> a time since that's all that each FETCH statement is asking for?
It's sending one row at a time. You could improve that by asking
for more than one row ("FETCH n" not just "FETCH"), but you'd have
to iterate over the rows returned by each command.
Another alternative is to forget about using a cursor, tell the
server to just execute the query, but use libpq's "chunked results"
mode to process rows before all of the query result has arrived.
See
https://www.postgresql.org/docs/current/libpq-single-row-mode.html
> I have done extensive searching to try and find the definitive answer to
> this. The searches indicate that libpq supports the concept of a
> client-side cursor, where it has a cache of rows sent by the server and
> uses that cache to perform each FETCH, but I'd like to be able to verify
> whether this is true or not.
Don't know where you read that, but it's not true of libpq. There
might be other Postgres client libraries that can do that.
regards, tom lane
Thanks for the quick reply Tom.
Let me give you a bit of background info:
Oracle has a COBOL pre-processor for embedded SQL, and it supports an option named PREFETCH:
The precompiler option PREFETCH allows for more efficient queries by pre-fetching rows. This decreases the number of server round-trips needed and reduces memory required. The number of rows set by the PREFETCH option value in a configuration file or on the command line is used for all queries involving explicit cursors, subject to the standard precedence rules. When used inline, the PREFETCH option must precede any of these cursor statements: • EXEC SQL OPEN cursor • EXEC SQL OPEN cursor USING host_var_list • EXEC SQL OPEN cursor USING DESCRIPTOR desc_name When an OPEN is executed, the value of PREFETCH gives the number of rows to be prefetched when the query is executed. You can set the value from 0 (no pre-fetching) to 9999. The default value is 1.
The IBM compiler for COBOL on Linux x86 recently added support for PGSQL (as a co-processor, integrated into the compiler). We have a business partner with a COBOL application that contains a large number of single-row FETCH statements, and they've noticed a significant performance degradation when switching from Oracle to PGSQL, and it's almost certainly because they're using that Oracle option PREFETCH. They are asserting that it's the large amount of client-server exchanges to retrieve one row at a time that is the problem in PGSQL.
As you can imagine, they're not wanting to rework their application(s) in order to achieve the same performance - they want the single-row FETCH to be as performant as Oracle. And I'm sure this same stance would be adopted by other users of the compiler who want to use PGSQL as well. They have loops in their programs which FETCH rows one at a time. I was hoping that the "chunked mode" you mentioned would achieve the equivalent of a client-side cursor, but your answer indicates that it won't.
The IBM co-processor is a rewrite of ECPG, and therefore uses libecpg at runtime, which of course uses libpq.
Do you have any further thoughts re: addressing this requirement (client-side cursor support)? Is this a requirement that the PGSQL community would consider implementing in libpq? Without it, it seems like a huge impediment to adoption of PGSQL for COBOL applications, and as you probably know, COBOL applications are still very pervasive. (C programs using ECPG would of course hit the same issue, so it's not just a COBOL-specific problem.)
Thanks,
Tim
On Wed, Dec 3, 2025 at 1:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Tim Fors <tim4stheenchanter@gmail.com> writes:
> I have a simple C program using libpq V18 to FETCH one row at a time (see
> below). Does the server send multiple rows to the client where they are
> cached for satisfying the FETCH statements, or does it just send one row at
> a time since that's all that each FETCH statement is asking for?
It's sending one row at a time. You could improve that by asking
for more than one row ("FETCH n" not just "FETCH"), but you'd have
to iterate over the rows returned by each command.
Another alternative is to forget about using a cursor, tell the
server to just execute the query, but use libpq's "chunked results"
mode to process rows before all of the query result has arrived.
See
https://www.postgresql.org/docs/current/libpq-single-row-mode.html
> I have done extensive searching to try and find the definitive answer to
> this. The searches indicate that libpq supports the concept of a
> client-side cursor, where it has a cache of rows sent by the server and
> uses that cache to perform each FETCH, but I'd like to be able to verify
> whether this is true or not.
Don't know where you read that, but it's not true of libpq. There
might be other Postgres client libraries that can do that.
regards, tom lane
On Wed, 2025-12-03 at 14:45 -0500, Tim Fors wrote: > Do you have any further thoughts re: addressing this requirement > (client-side cursor support)? Is this a requirement that the PGSQL > community would consider implementing in libpq? Without it, it seems > like a huge impediment to adoption of PGSQL for COBOL applications, > and as you probably know, COBOL applications are still very > pervasive. (C programs using ECPG would of course hit the same issue, > so it's not just a COBOL-specific problem.) Sounds to me like this ought to be implemented in ECPG and/or its COBOL equivalent, not in libpq? That's where the "Embedded SQL" is implemented after all, and also the equivalent of where Oracle implements its PREFETCH, I think? And I suppose nobody would be opposed to such improvements, if someone contributes them or pays for their development... -- Jan Claeys (please don't CC me when replying to the list)