David Hartwig <daybee@bellatlantic.net> writes:
> We did not (and still do not) like the
> way libpq consumes the entire query result into front-end memory before
> returning control to the application. This make large queries slow and
> sometimes impossible.
With the new PQsendQuery/PQgetResult API, it would not take very much
work to address this objection. The application could set a limit on
the number of tuples to be returned per PGresult (maybe about 100 would
reasonable), and then use a loop calling PQgetResult to fetch all the
tuples of a large query. We'd probably want to add a new boolean flag
"end of query/not end of query" to PGresult so that one could use this
feature even when sending multiple SELECTs in one sendQuery.
Rather than make the limit a persistent setting of a PGconn object,
I'd be inclined to add it as a parameter to PQgetResult (I think it's
not too late to change the calling sequence for PQgetResult, since it
hasn't yet seen the light of day in any formal release). Otherwise
PQexec might start behaving unexpectedly in an application that contains
a mixture of old-style PQexec calls and new-style PQsendQuery/getResult.
If the memory consumption issue is a hot button for anyone who is using
(or would use) libpq, I'll see if I can squeeze this in for 6.4. I assume
the psqlodbc guys will stick to their own code anyway.
regards, tom lane