Обсуждение: Revised proposal for libpq and FE/BE protocol changes
Here is a revised proposal that takes into account the discussions
of the last few days. Any comments?
I propose to revise libpq and modify the frontend/backend protocol
to provide the following benefits:
* Provide a clean way of reading multiple results from a single query
string. Among other things, this solves the problem of allowing a
single query to return several result sets with different descriptors.
* Allow a frontend to perform other work while awaiting the result of
a query.
* Add the ability to cancel queries in progress.
* Eliminate the need for frontends to issue dummy queries in order
to detect NOTIFY responses.
* Eliminate the need for libpq to issue dummy queries internally
to determine when a query is complete.
We can't break existing code for this, so the behavior of PQexec()
can't change. Instead, I propose new functions to add to the API.
Internally, PQexec will be reimplemented in terms of these new
functions, but old applications shouldn't notice any difference.
The new functions are:
bool PQsendQuery (PGconn *conn, const char *query);
Submits a query without waiting for the result. Returns TRUE if the
query has been successfully dispatched, otherwise FALSE (in the FALSE
case, an error message is left in conn->errorMessage).
PGresult* PQgetResult (PGconn *conn);
Waits for input from the backend, and consumes input until (a) a result is
available, (b) the current query is over, or (c) a copy in/out operation
is detected. NULL is returned if the query is over; in all other cases a
suitable PGresult is returned (which the caller must eventually free).
Note that no actual "wait" will occur if the necessary input has already
been consumed; see below.
bool PQisBusy (PGconn *conn);
Returns TRUE if a query operation is busy (that is, a call to PQgetResult
would block waiting for more input). Returns FALSE if PQgetResult would
return immediately.
void PQconsumeInput (PGconn *conn);
This can be called at any time to check for and process new input from
the backend. It returns no status indication, but after calling it
the application can use PQisBusy() and/or PQnotifies() to see if a query
was completed or a NOTIFY message arrived. This function will never wait
for more input to arrive.
int PQsocket (PGconn *conn);
Returns the Unix file descriptor for the socket connection to the backend,
or -1 if there is no open connection. This is a violation of modularity,
of course, but there is no alternative: an application that needs
asynchronous execution needs to be able to use select() to wait for input
from either the backend or any other input streams it may have. To use
select() the underlying socket must be made visible.
PGnotify *PQnotifies (PGconn *conn);
This function doesn't change; we just observe that notifications may
become available as a side effect of executing either PQgetResult() or
PQconsumeInput(), not just PQexec().
void PQrequestCancel (PGconn *conn);
Issues a cancel request if possible. There is no direct way to tell whether
this has any effect ... see discussion below.
Discussion:
An application can continue to use PQexec() as before, and notice
very little difference in behavior.
Applications that want to be able to handle multiple results from a
single query should replace PQexec calls with logic like this:
// Submit the query
if (! PQsendQuery(conn, query))
reportTheError();
// Wait for and process result(s)
while ((result = PQgetResult(conn)) != NULL) {
switch (PQresultStatus(result)) {
... process result, for example:
case PGRES_COPY_IN:
// ... copy data here ...
if (PQendcopy(conn))
reportTheError();
break;
...
}
PQclear(result);
}
// When fall out of loop, we're done and ready for a new query
Note that PQgetResult will always report errors by returning a PGresult
with status PGRES_NONFATAL_ERROR or PGRES_FATAL_ERROR, not by returning
NULL (since NULL implies non-error termination of the processing loop).
PQexec() will be implemented as follows:
if (! PQsendQuery(conn, query))
return makeEmptyPGresult(conn, PGRES_FATAL_ERROR);
lastResult = NULL;
while ((result = PQgetResult(conn)) != NULL) {
PQclear(lastResult);
lastResult = result;
}
return lastResult;
This maintains the current behavior that the last result of a series
of commands is returned by PQexec. (The old implementation is only
capable of doing that correctly in a limited set of cases, but in the
cases where it behaves usefully at all, that's how it behaves.)
There is a small difference in behavior, which is that PQexec will now
return a PGresult with status PGRES_FATAL_ERROR in cases where the old
implementation would just have returned NULL (and set conn->errorMessage).
However, any correctly coded application should handle this the same way.
In the above examples, the frontend application is still synchronous: it
blocks while waiting for the backend to reply to a query. This is often
undesirable, since the application may have other work to do, such as
responding to user input. Applications can now handle that by using
PQisBusy and PQconsumeInput along with PQsendQuery and PQgetResult.
The general idea is that the application's main loop will use select()
to wait for input (from either the backend or its other input sources).
When select() indicates that input is pending from the backend, the app
will call PQconsumeInput, followed by checking PQisBusy and/or PQnotifies
to see what has happened. If PQisBusy returns FALSE then PQgetResult
can safely be called to obtain and process a result without blocking.
Note also that NOTIFY messages can arrive asynchronously from the backend.
They can be detected *without issuing a query* by calling PQconsumeInput
followed by PQnotifies. I expect a lot of people will build "partially
async" applications that detect notifies this way but still do all their
queries through PQexec (or better, PQsendQuery followed by a synchronous
PQgetResult loop). This compromise allows notifies to be detected without
wasting time by issuing null queries, yet the basic logic of issuing a
series of queries remains simple.
Finally, since the application can retain control while waiting for a
query response, it becomes meaningful to try to cancel a query in progress.
This is done by calling PQrequestCancel(). Note that PQrequestCancel()
may not have any effect --- if there is no query in progress, or if the
backend has already finished the query, then it *will* have no effect.
The application must continue to follow the result-reading protocol after
issuing a cancel request. If the cancel is successful, its effect will be
to cause the current query to fail and return an error message.
PROTOCOL CHANGES:
We should change the protocol version number to 2.0.
It would be possible for the backend to continue to support 1.0 clients,
if you think it's worth the trouble to do so.
1. New message type:
Command Done
Byte1('Z')
The backend will emit this message at completion of processing of every
command string, just before it resumes waiting for frontend input.
This change eliminates libpq's current hack of issuing empty queries to
see whether the backend is done. Note that 'Z' must be emitted after
*every* query or function invocation, no matter how it terminated.
2. The RowDescription ('T') message is extended by adding a new value
for each field. Just after the type-size value, there will now be
an int16 "atttypmod" value. (Would someone provide text specifying
exactly what this value means?) libpq will store this value in
a new "adtmod" field of PGresAttDesc structs.
3. The "Start Copy In" response message is changed from 'D' to 'G',
and the "Start Copy Out" response message is changed from 'B' to 'H'.
These changes eliminate potential confusion with the data row messages,
which also have message codes 'D' and 'B'.
4. The frontend may request cancellation of the current query by sending
a single byte of OOB (out-of-band) data. The contents of the data byte
are irrelevant, since the cancellation will be triggered by the associated
signal and not by the data itself. (But we should probably specify that
the byte be zero, in case we later think of a reason to have different
kinds of OOB messages.) There is no specific reply to this message.
If the backend does cancel a query, the query terminates with an ordinary
error message indicating that the query was cancelled.
regards, tom lane
> 2. The RowDescription ('T') message is extended by adding a new value
> for each field. Just after the type-size value, there will now be
> an int16 "atttypmod" value. (Would someone provide text specifying
> exactly what this value means?) libpq will store this value in
> a new "adtmod" field of PGresAttDesc structs.
From src/include/catalog/pg_attribute.h:
/*
* atttypmod records type-specific modifications supplied at table
* creation time, and passes it to input and output functions as the
* third argument.
*/
Currently only used for char() and varchar(), and includes a 4-byte
header.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
Tom Lane wrote: > > Here is a revised proposal that takes into account the discussions > of the last few days. Any comments? Just one at the end [snip] > 4. The frontend may request cancellation of the current query by sending > a single byte of OOB (out-of-band) data. The contents of the data byte > are irrelevant, since the cancellation will be triggered by the associated > signal and not by the data itself. (But we should probably specify that > the byte be zero, in case we later think of a reason to have different > kinds of OOB messages.) There is no specific reply to this message. > If the backend does cancel a query, the query terminates with an ordinary > error message indicating that the query was cancelled. You didn't come right out and say it, but are you intending to support multiple queries within a connection? I gather not. Not that I'm suggesting that this be done, as it seems this would complicate the user's application and the backend. With only one possible OOB message, you can't tell it which query to cancel. Ocie Mitchell
Re: [INTERFACES] Revised proposal for libpq and FE/BE protocol changes
От
watts@humbug.antnet.com
Дата:
I suggest the application already has fork or fork/exec to
implement an asynchronous design. Does that also keep the
socket out of the application's domain?
Bob
watts@humbug.antnet.com
Received: from hub.org (hub.org [209.47.148.200])
by humbug.antnet.com (8.8.5/8.8.5) with ESMTP id LAA21503
for <watts@humbug.antnet.com>; Tue, 28 Apr 1998 11:28:48 -0500 (CDT)
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id MAA01511; Tue, 28 Apr 1998 12:23:18
-0400(EDT)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 28 Apr 1998 12:23:16 -0400 (EDT)
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id MAA01498 for pgsql-interfaces-outgoing; Tue, 28 Apr
199812:23:09 -0400 (EDT)
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id MAA01401; Tue,
28Apr 1998 12:22:04 -0400 (EDT)
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1])
by sss.sss.pgh.pa.us (8.8.5/8.8.5) with ESMTP id MAA07043;
Tue, 28 Apr 1998 12:21:56 -0400 (EDT)
To: pgsql-hackers@postgreSQL.org, pgsql-interfaces@postgreSQL.org
Subject: [INTERFACES] Revised proposal for libpq and FE/BE protocol changes
Date: Tue, 28 Apr 1998 12:21:55 -0400
Message-ID: <7040.893780515@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
Sender: owner-pgsql-interfaces@hub.org
Precedence: bulk
Here is a revised proposal that takes into account the discussions
of the last few days. Any comments?
I propose to revise libpq and modify the frontend/backend protocol
to provide the following benefits:
* Provide a clean way of reading multiple results from a single query
string. Among other things, this solves the problem of allowing a
single query to return several result sets with different descriptors.
* Allow a frontend to perform other work while awaiting the result of
a query.
* Add the ability to cancel queries in progress.
* Eliminate the need for frontends to issue dummy queries in order
to detect NOTIFY responses.
* Eliminate the need for libpq to issue dummy queries internally
to determine when a query is complete.
We can't break existing code for this, so the behavior of PQexec()
can't change. Instead, I propose new functions to add to the API.
Internally, PQexec will be reimplemented in terms of these new
functions, but old applications shouldn't notice any difference.
The new functions are:
bool PQsendQuery (PGconn *conn, const char *query);
Submits a query without waiting for the result. Returns TRUE if the
query has been successfully dispatched, otherwise FALSE (in the FALSE
case, an error message is left in conn->errorMessage).
PGresult* PQgetResult (PGconn *conn);
Waits for input from the backend, and consumes input until (a) a result is
available, (b) the current query is over, or (c) a copy in/out operation
is detected. NULL is returned if the query is over; in all other cases a
suitable PGresult is returned (which the caller must eventually free).
Note that no actual "wait" will occur if the necessary input has already
been consumed; see below.
bool PQisBusy (PGconn *conn);
Returns TRUE if a query operation is busy (that is, a call to PQgetResult
would block waiting for more input). Returns FALSE if PQgetResult would
return immediately.
void PQconsumeInput (PGconn *conn);
This can be called at any time to check for and process new input from
the backend. It returns no status indication, but after calling it
the application can use PQisBusy() and/or PQnotifies() to see if a query
was completed or a NOTIFY message arrived. This function will never wait
for more input to arrive.
int PQsocket (PGconn *conn);
Returns the Unix file descriptor for the socket connection to the backend,
or -1 if there is no open connection. This is a violation of modularity,
of course, but there is no alternative: an application that needs
asynchronous execution needs to be able to use select() to wait for input
from either the backend or any other input streams it may have. To use
select() the underlying socket must be made visible.
PGnotify *PQnotifies (PGconn *conn);
This function doesn't change; we just observe that notifications may
become available as a side effect of executing either PQgetResult() or
PQconsumeInput(), not just PQexec().
void PQrequestCancel (PGconn *conn);
Issues a cancel request if possible. There is no direct way to tell whether
this has any effect ... see discussion below.
Discussion:
An application can continue to use PQexec() as before, and notice
very little difference in behavior.
Applications that want to be able to handle multiple results from a
single query should replace PQexec calls with logic like this:
// Submit the query
if (! PQsendQuery(conn, query))
reportTheError();
// Wait for and process result(s)
while ((result = PQgetResult(conn)) != NULL) {
switch (PQresultStatus(result)) {
... process result, for example:
case PGRES_COPY_IN:
// ... copy data here ...
if (PQendcopy(conn))
reportTheError();
break;
...
}
PQclear(result);
}
// When fall out of loop, we're done and ready for a new query
Note that PQgetResult will always report errors by returning a PGresult
with status PGRES_NONFATAL_ERROR or PGRES_FATAL_ERROR, not by returning
NULL (since NULL implies non-error termination of the processing loop).
PQexec() will be implemented as follows:
if (! PQsendQuery(conn, query))
return makeEmptyPGresult(conn, PGRES_FATAL_ERROR);
lastResult = NULL;
while ((result = PQgetResult(conn)) != NULL) {
PQclear(lastResult);
lastResult = result;
}
return lastResult;
This maintains the current behavior that the last result of a series
of commands is returned by PQexec. (The old implementation is only
capable of doing that correctly in a limited set of cases, but in the
cases where it behaves usefully at all, that's how it behaves.)
There is a small difference in behavior, which is that PQexec will now
return a PGresult with status PGRES_FATAL_ERROR in cases where the old
implementation would just have returned NULL (and set conn->errorMessage).
However, any correctly coded application should handle this the same way.
In the above examples, the frontend application is still synchronous: it
blocks while waiting for the backend to reply to a query. This is often
undesirable, since the application may have other work to do, such as
responding to user input. Applications can now handle that by using
PQisBusy and PQconsumeInput along with PQsendQuery and PQgetResult.
The general idea is that the application's main loop will use select()
to wait for input (from either the backend or its other input sources).
When select() indicates that input is pending from the backend, the app
will call PQconsumeInput, followed by checking PQisBusy and/or PQnotifies
to see what has happened. If PQisBusy returns FALSE then PQgetResult
can safely be called to obtain and process a result without blocking.
Note also that NOTIFY messages can arrive asynchronously from the backend.
They can be detected *without issuing a query* by calling PQconsumeInput
followed by PQnotifies. I expect a lot of people will build "partially
async" applications that detect notifies this way but still do all their
queries through PQexec (or better, PQsendQuery followed by a synchronous
PQgetResult loop). This compromise allows notifies to be detected without
wasting time by issuing null queries, yet the basic logic of issuing a
series of queries remains simple.
Finally, since the application can retain control while waiting for a
query response, it becomes meaningful to try to cancel a query in progress.
This is done by calling PQrequestCancel(). Note that PQrequestCancel()
may not have any effect --- if there is no query in progress, or if the
backend has already finished the query, then it *will* have no effect.
The application must continue to follow the result-reading protocol after
issuing a cancel request. If the cancel is successful, its effect will be
to cause the current query to fail and return an error message.
PROTOCOL CHANGES:
We should change the protocol version number to 2.0.
It would be possible for the backend to continue to support 1.0 clients,
if you think it's worth the trouble to do so.
1. New message type:
Command Done
Byte1('Z')
The backend will emit this message at completion of processing of every
command string, just before it resumes waiting for frontend input.
This change eliminates libpq's current hack of issuing empty queries to
see whether the backend is done. Note that 'Z' must be emitted after
*every* query or function invocation, no matter how it terminated.
2. The RowDescription ('T') message is extended by adding a new value
for each field. Just after the type-size value, there will now be
an int16 "atttypmod" value. (Would someone provide text specifying
exactly what this value means?) libpq will store this value in
a new "adtmod" field of PGresAttDesc structs.
3. The "Start Copy In" response message is changed from 'D' to 'G',
and the "Start Copy Out" response message is changed from 'B' to 'H'.
These changes eliminate potential confusion with the data row messages,
which also have message codes 'D' and 'B'.
4. The frontend may request cancellation of the current query by sending
a single byte of OOB (out-of-band) data. The contents of the data byte
are irrelevant, since the cancellation will be triggered by the associated
signal and not by the data itself. (But we should probably specify that
the byte be zero, in case we later think of a reason to have different
kinds of OOB messages.) There is no specific reply to this message.
If the backend does cancel a query, the query terminates with an ordinary
error message indicating that the query was cancelled.
regards, tom lane
Re: [HACKERS] Revised proposal for libpq and FE/BE protocol changes
От
dg@illustra.com (David Gould)
Дата:
>
> Tom Lane wrote:
> >
> > Here is a revised proposal that takes into account the discussions
> > of the last few days. Any comments?
>
> Just one at the end
>
> [snip]
>
> > 4. The frontend may request cancellation of the current query by sending
> > a single byte of OOB (out-of-band) data. The contents of the data byte
> > are irrelevant, since the cancellation will be triggered by the associated
> > signal and not by the data itself. (But we should probably specify that
> > the byte be zero, in case we later think of a reason to have different
> > kinds of OOB messages.) There is no specific reply to this message.
> > If the backend does cancel a query, the query terminates with an ordinary
> > error message indicating that the query was cancelled.
>
> You didn't come right out and say it, but are you intending to support
> multiple queries within a connection? I gather not. Not that I'm
> suggesting that this be done, as it seems this would complicate the
> user's application and the backend. With only one possible OOB
> message, you can't tell it which query to cancel.
>
> Ocie Mitchell
Waves hand wildly... I know, I know!
All of them!
-dg
David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"(Windows NT) version 5.0 will build on a proven system architecture
and incorporate tens of thousands of bug fixes from version 4.0."
-- <http://www.microsoft.com/y2k.asp?A=7&B=5>
Tom Lane writes: > I propose to revise libpq and modify the frontend/backend protocol > to provide the following benefits: > * Provide a clean way of reading multiple results from a single query > string. Among other things, this solves the problem of allowing a > single query to return several result sets with different descriptors. Does this mean I can read in a complete C array with one call? I mean something like this: char emp_name[10][10]; exec sql select name into :emp_name from emp; But then I didn't see anything like this in your examples. Do I have to iterate using PQgetResult then? Michael -- Dr. Michael Meskes, Project-Manager | topsystem Systemhaus GmbH meskes@topsystem.de | Europark A2, Adenauerstr. 20 meskes@debian.org | 52146 Wuerselen Go SF49ers! Go Rhein Fire! | Tel: (+49) 2405/4670-44 Use Debian GNU/Linux! | Fax: (+49) 2405/4670-10
ocie@paracel.com writes:
> You didn't come right out and say it, but are you intending to support
> multiple queries within a connection? I gather not. Not that I'm
> suggesting that this be done, as it seems this would complicate the
> user's application and the backend. With only one possible OOB
> message, you can't tell it which query to cancel.
That was something I asked about a few days ago, and didn't get any
responses suggesting that anyone thought it was likely to happen.
We would need wholesale changes everywhere in the protocol to support
concurrent queries: answers and errors coming back would have to be
tagged to indicate which query they apply to. The lack of a tag in
the cancel message isn't the controlling factor.
In the current system architecture, much the easiest way to execute
concurrent queries is to open up more than one connection. There's
nothing that says a frontend process can't fire up multiple backend
processes. I think this is probably sufficient, because I don't
foresee such a thing becoming really popular anyway.
regards, tom lane
Michael Meskes <meskes@topsystem.de> writes:
> Does this mean I can read in a complete C array with one call? I mean
> something like this:
> char emp_name[10][10];
> exec sql select name into :emp_name from emp;
As far as I know that works now; or at least, if it doesn't work it's
a limitation of the embedded-SQL interface, and not anything that has
to do with libpq or the fe/be protocol.
A "result" in libpq's terms is the result of a single SQL command.
The result of a successful query, for example, is typically multiple
rows of data. You only need a PQgetResult loop if (a) you send a
query string that contains several commands, or (b) you issue a
query whose answer contains more than one kind of tuple.
regards, tom lane
watts@humbug.antnet.com writes:
> I suggest the application already has fork or fork/exec to
> implement an asynchronous design.
True, if you don't mind assuming you have threads then you could
dedicate one thread to blocking in libpq while your other threads manage
your user interface and so forth. But most of these revisions would
still be useful in that situation. The current libpq does not cope well
with query strings containing multiple commands; it doesn't cope at all
with queries that return more than one type of tuple; it requires dummy
queries (wasting both processing time and network bandwidth) to check
for NOTIFY messages; and so forth. None of those problems can be solved
just by moving calls to libpq into a separate thread.
regards, tom lane
> That was something I asked about a few days ago, and didn't get any > responses suggesting that anyone thought it was likely to happen. > > We would need wholesale changes everywhere in the protocol to support > concurrent queries: answers and errors coming back would have to be > tagged to indicate which query they apply to. The lack of a tag in > the cancel message isn't the controlling factor. > > In the current system architecture, much the easiest way to execute > concurrent queries is to open up more than one connection. There's > nothing that says a frontend process can't fire up multiple backend > processes. I think this is probably sufficient, because I don't > foresee such a thing becoming really popular anyway. If we can remove the exec() in 6.4, that will make backend startup even quicker. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)
Tom Lane wrote:
>
> PROTOCOL CHANGES:
>
> We should change the protocol version number to 2.0.
> It would be possible for the backend to continue to support 1.0 clients,
> if you think it's worth the trouble to do so.
Or 1.1? The changes don't seem too traumatic. Either way, maintaining
support for 1.0 is important as not all of us use libpq and we need time
to catch up. Also we don't want to put barriers in the way of companies
like Openlink who seem willing to provide support for PostgreSQL in
commercial products.
> 1. New message type:
>
> Command Done
> Byte1('Z')
>
> The backend will emit this message at completion of processing of every
> command string, just before it resumes waiting for frontend input.
> This change eliminates libpq's current hack of issuing empty queries to
> see whether the backend is done. Note that 'Z' must be emitted after
> *every* query or function invocation, no matter how it terminated.
The completion response already does this for successful queries, and
the error response for unsuccessful ones. I came to the conclusion (but
not with absolute certainty) a while back that the empty query hack was
needed for an old feature of the backend that is no longer there. From
looking at a dump of the data between psql and the backend for 6.3.2 I
don't think that those empty queries are issued any more. I have
implemented a pure Tcl frontend that doesn't issue them and I haven't
seen any problems.
The exception to the above is the single empty query sent immediately
after the frontend has been successfully authenticated. This is useful
because it has the side effect of checking that the user has privileges
against the particular database - it is better to do this as part of the
session set up rather than the first real query which may be some time
later.
> 3. The "Start Copy In" response message is changed from 'D' to 'G',
> and the "Start Copy Out" response message is changed from 'B' to 'H'.
> These changes eliminate potential confusion with the data row messages,
> which also have message codes 'D' and 'B'.
The context means there should be no confusion - but if the protocol is
being changed anyway then it makes sense to do this.
Phil
Phil Thompson <phil@river-bank.demon.co.uk> writes:
> Tom Lane wrote:
>> We should change the protocol version number to 2.0.
>> It would be possible for the backend to continue to support 1.0 clients,
>> if you think it's worth the trouble to do so.
> Or 1.1? The changes don't seem too traumatic.
Well, pqcomm.h says that an incompatible change should have a new major
version number, and minor though these changes be, they *are*
incompatible.
> Either way, maintaining support for 1.0 is important as not all of us
> use libpq and we need time to catch up.
No argument from me. It shouldn't be hard to emit the new stuff
conditionally.
>> Command Done
>> Byte1('Z')
> The completion response already does this for successful queries, and
> the error response for unsuccessful ones.
You missed the point: it is possible to send more than one SQL command
in a single query string. The reason that libpq sends empty queries is
to determine whether the backend is actually done processing the string.
I suppose we could instead try to make libpq smart enough to parse the
string it's sending and determine how many responses to expect ... but
it seems much easier and more robust to have the backend tell us when
it's done.
> From looking at a dump of the data between psql and the backend for
> 6.3.2 I don't think that those empty queries are issued any more.
> I have implemented a pure Tcl frontend that doesn't issue them and I
> haven't seen any problems.
You didn't exercise the cases where they are sent. A command that
generates a "C" response without tuple data is needed to make libpq
insert an empty query at the moment. The code is a horrible kluge,
because it only works for cases like
set geqo to 'off'; show datestyle; select * from table;
and not for, say,
select * from table1; select * from table2;
psql masks the problem because it splits up your input into separate
commands and hands them to libpq one at a time. A dumber UI is needed
to exhibit the trouble. (We should be able to rip out the
command-splitting code from psql after making this change, BTW.
I think it'll be better to have neither psql nor libpq know much of
anything about SQL syntax.)
> The exception to the above is the single empty query sent immediately
> after the frontend has been successfully authenticated. This is useful
Right. I didn't plan to remove that one.
regards, tom lane
> > Tom Lane wrote: > > > > PROTOCOL CHANGES: > > > > We should change the protocol version number to 2.0. > > It would be possible for the backend to continue to support 1.0 clients, > > if you think it's worth the trouble to do so. > > Or 1.1? The changes don't seem too traumatic. Either way, maintaining > support for 1.0 is important as not all of us use libpq and we need time > to catch up. Also we don't want to put barriers in the way of companies > like Openlink who seem willing to provide support for PostgreSQL in > commercial products. Yes, but there will be a month for people to get their third-part stuff changed, and the changes are pretty straight-forward. Having support for both in the backend/frontend is going to make that code more difficult. If it was only a small change, we could keep it compatable, but it seems it would be best to just announce it early on. People can start testing their new drivers long before the beta period begins. Also, we are making this change well in advance of the beta, so I hope they would have enough time to make the transition. > > The backend will emit this message at completion of processing of every > > command string, just before it resumes waiting for frontend input. > > This change eliminates libpq's current hack of issuing empty queries to > > see whether the backend is done. Note that 'Z' must be emitted after > > *every* query or function invocation, no matter how it terminated. > > The completion response already does this for successful queries, and > the error response for unsuccessful ones. I came to the conclusion (but > not with absolute certainty) a while back that the empty query hack was > needed for an old feature of the backend that is no longer there. From > looking at a dump of the data between psql and the backend for 6.3.2 I > don't think that those empty queries are issued any more. I have > implemented a pure Tcl frontend that doesn't issue them and I haven't > seen any problems. > > The exception to the above is the single empty query sent immediately > after the frontend has been successfully authenticated. This is useful > because it has the side effect of checking that the user has privileges > against the particular database - it is better to do this as part of the > session set up rather than the first real query which may be some time > later. Good insight on the libpq interface. I think we need the new return code because of the possibility of multiple results from the backend. In the old code, without the empty query, doesn't a query with multiple statements cause the send/return results to get out of sync. > > 3. The "Start Copy In" response message is changed from 'D' to 'G', > > and the "Start Copy Out" response message is changed from 'B' to 'H'. > > These changes eliminate potential confusion with the data row messages, > > which also have message codes 'D' and 'B'. > > The context means there should be no confusion - but if the protocol is > being changed anyway then it makes sense to do this. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)
Tom Lane wrote:
>
> Phil Thompson <phil@river-bank.demon.co.uk> writes:
> > Tom Lane wrote:
> >> We should change the protocol version number to 2.0.
> >> It would be possible for the backend to continue to support 1.0 clients,
> >> if you think it's worth the trouble to do so.
>
> > Or 1.1? The changes don't seem too traumatic.
>
> Well, pqcomm.h says that an incompatible change should have a new major
> version number, and minor though these changes be, they *are*
> incompatible.
Err...good point :)
> >> Command Done
> >> Byte1('Z')
>
> > The completion response already does this for successful queries, and
> > the error response for unsuccessful ones.
>
> You missed the point:
I've misunderstood the protocol - and the protocol specification is
therefore wrong (or at least incomplete) in this respect. Do you want
to fix the spec and include your enhancements or shall I?
Phil
Bruce Momjian wrote: > > Either way, maintaining > > support for 1.0 is important as not all of us use libpq and we need time > > to catch up. Also we don't want to put barriers in the way of companies > > like Openlink who seem willing to provide support for PostgreSQL in > > commercial products. > > Yes, but there will be a month for people to get their third-part stuff > changed, and the changes are pretty straight-forward. Having support > for both in the backend/frontend is going to make that code more > difficult. I agree it will be easy enough for most of us, but may be less so for companies that traditionally don't release often. Although I don't use Openlink's software and can't comment on whether it's any good (or if anybody actually uses it), I take it as a compliment to PostgreSQL that a commercial organisation is willing to provide some support for it. Not maintaining backwards compatibility for at least some time isn't going to encourage them to continue that support. Phil
Phil Thompson <phil@river-bank.demon.co.uk> writes:
> I've misunderstood the protocol - and the protocol specification is
> therefore wrong (or at least incomplete) in this respect. Do you want
> to fix the spec and include your enhancements or shall I?
Yes, there are some things I thought were wrong in the programmer's guide
chapter about the FE/BE protocol. I'd be happy to submit revised text.
I haven't paid any attention yet to how the documentation is handled.
Is the stuff in the distribution under doc/src/sgml considered the
editable master text, or is it generated from some other format?
Do I need to subscribe to pgsql-docs to find out what to do? :-)
regards, tom lane
Re: [INTERFACES] Re: [HACKERS] Revised proposal for libpq and FE/BE protocol changes
От
"Thomas G. Lockhart"
Дата:
Tom Lane wrote:
>
> Phil Thompson <phil@river-bank.demon.co.uk> writes:
> > I've misunderstood the protocol - and the protocol specification is
> > therefore wrong (or at least incomplete) in this respect. Do you want
> > to fix the spec and include your enhancements or shall I?
>
> Yes, there are some things I thought were wrong in the programmer's guide
> chapter about the FE/BE protocol. I'd be happy to submit revised text.
>
> I haven't paid any attention yet to how the documentation is handled.
> Is the stuff in the distribution under doc/src/sgml considered the
> editable master text, or is it generated from some other format?
Yes, doc/src/sgml/*.sgml is the editable format. Phil's writeup is in
protocol.sgml and that is all that would need to be touched. Then submit
the patches (or a replacement file since you are the only one editing
that at the moment) and we'll snarf them up and merge.
The SGML markup is not the prettiest, but don't be intimidated/annoyed
by it. Especially if you are making incremental changes, I would guess
that you will see how to cut and paste using the existing markup tags in
the file. Let us know if you have any trouble with it. We can fix markup
problems after you post changes too...
> Do I need to subscribe to pgsql-docs to find out what to do? :-)
Only if you want to keep writing :)
- Tom
Re: [HACKERS] Revised proposal for libpq and FE/BE protocol changes
От
dg@illustra.com (David Gould)
Дата:
Bruce Momjian wrote:
> >
> > Tom Lane wrote:
> > >
> > > PROTOCOL CHANGES:
> > >
> > > We should change the protocol version number to 2.0.
> > > It would be possible for the backend to continue to support 1.0 clients,
> > > if you think it's worth the trouble to do so.
> >
> > Or 1.1? The changes don't seem too traumatic. Either way, maintaining
> > support for 1.0 is important as not all of us use libpq and we need time
> > to catch up. Also we don't want to put barriers in the way of companies
> > like Openlink who seem willing to provide support for PostgreSQL in
> > commercial products.
>
> Yes, but there will be a month for people to get their third-part stuff
> changed, and the changes are pretty straight-forward. Having support
> for both in the backend/frontend is going to make that code more
> difficult.
>
> If it was only a small change, we could keep it compatable, but it seems
> it would be best to just announce it early on. People can start testing
> their new drivers long before the beta period begins.
>
> Also, we are making this change well in advance of the beta, so I hope
> they would have enough time to make the transition.
I know this is old old discussion, so "shut up, we're done with it" is a
fine answer...
But, I think maintaining compatibility with 1.0 is important. If we expect
people to really use this software to build real applications, then we
cannot expect them to be interested in revising or even recompiling their
applications.
For example a web development consulting house. They build shopping cart
or other database using web sites for their clients. They do not want to
have to go to each of their customers (say hundreds of sites) and recompile
everything just to take advantage of a new server that happens to fix some
bugs they needed fixed. They also don't want to reload the databases or
otherwise get involved in upgrade issues. And, their clients don't want
to pay for this either.
Database customers at least in the commercial world can be incredibly
conservative. It is not at all uncommon to have large sites running DBMS
engines that are three major releases (ie, well over three years) old.
Once they get an app working, they really don't want anything to change.
-dg
David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
and someone who knows less will correct me if I'm right."
--David Palmer (palmer@tybalt.caltech.edu)
> Database customers at least in the commercial world can be incredibly > conservative. It is not at all uncommon to have large sites running DBMS > engines that are three major releases (ie, well over three years) old. > Once they get an app working, they really don't want anything to change. And, oh yea, we kept database compatability, thanks to Tom Lane. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)
> Database customers at least in the commercial world can be incredibly > conservative. It is not at all uncommon to have large sites running DBMS > engines that are three major releases (ie, well over three years) old. > Once they get an app working, they really don't want anything to change. Yes, this is true. Their data is locked in Our database. And you can't just restart it like a PC OS or word processor. Database demands are much different. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)
dg@illustra.com (David Gould) writes:
> I know this is old old discussion, so "shut up, we're done with it" is a
> fine answer...
> But, I think maintaining compatibility with 1.0 is important.
We did maintain compatibility, in that the server will still talk to
a client that identifies itself as 1.0 in the initial handshake.
(Kudos to whoever had the foresight to put a protocol version number
in the startup message, BTW.)
As things are currently set up, however, a new compilation of libpq
will only know how to talk protocol 2.0, so it cannot be used to
talk to an old server. Are you concerned about that? I don't see
any easy way around it...
regards, tom lane
Tom, just wondering were we are with this. Can you update libpq.3? I
think until the sgml of the manual is converted, they are the most
current. I just made some cleanups there myself. Are the sgml sources
updated with the protocol changes?
Also, are these items completed? How about our cancel query key? I
think it is random/secure enough for our purposes. Can you make the
changes, or do you need changes from me?
---------------------------------------------------------------------------
> Here is a revised proposal that takes into account the discussions
> of the last few days. Any comments?
>
>
> I propose to revise libpq and modify the frontend/backend protocol
> to provide the following benefits:
> * Provide a clean way of reading multiple results from a single query
> string. Among other things, this solves the problem of allowing a
> single query to return several result sets with different descriptors.
> * Allow a frontend to perform other work while awaiting the result of
> a query.
> * Add the ability to cancel queries in progress.
> * Eliminate the need for frontends to issue dummy queries in order
> to detect NOTIFY responses.
> * Eliminate the need for libpq to issue dummy queries internally
> to determine when a query is complete.
>
> We can't break existing code for this, so the behavior of PQexec()
> can't change. Instead, I propose new functions to add to the API.
> Internally, PQexec will be reimplemented in terms of these new
> functions, but old applications shouldn't notice any difference.
>
>
> The new functions are:
>
> bool PQsendQuery (PGconn *conn, const char *query);
>
> Submits a query without waiting for the result. Returns TRUE if the
> query has been successfully dispatched, otherwise FALSE (in the FALSE
> case, an error message is left in conn->errorMessage).
>
> PGresult* PQgetResult (PGconn *conn);
>
> Waits for input from the backend, and consumes input until (a) a result is
> available, (b) the current query is over, or (c) a copy in/out operation
> is detected. NULL is returned if the query is over; in all other cases a
> suitable PGresult is returned (which the caller must eventually free).
> Note that no actual "wait" will occur if the necessary input has already
> been consumed; see below.
>
> bool PQisBusy (PGconn *conn);
>
> Returns TRUE if a query operation is busy (that is, a call to PQgetResult
> would block waiting for more input). Returns FALSE if PQgetResult would
> return immediately.
>
> void PQconsumeInput (PGconn *conn);
>
> This can be called at any time to check for and process new input from
> the backend. It returns no status indication, but after calling it
> the application can use PQisBusy() and/or PQnotifies() to see if a query
> was completed or a NOTIFY message arrived. This function will never wait
> for more input to arrive.
>
> int PQsocket (PGconn *conn);
>
> Returns the Unix file descriptor for the socket connection to the backend,
> or -1 if there is no open connection. This is a violation of modularity,
> of course, but there is no alternative: an application that needs
> asynchronous execution needs to be able to use select() to wait for input
> from either the backend or any other input streams it may have. To use
> select() the underlying socket must be made visible.
>
> PGnotify *PQnotifies (PGconn *conn);
>
> This function doesn't change; we just observe that notifications may
> become available as a side effect of executing either PQgetResult() or
> PQconsumeInput(), not just PQexec().
>
> void PQrequestCancel (PGconn *conn);
>
> Issues a cancel request if possible. There is no direct way to tell whether
> this has any effect ... see discussion below.
>
>
> Discussion:
>
> An application can continue to use PQexec() as before, and notice
> very little difference in behavior.
>
> Applications that want to be able to handle multiple results from a
> single query should replace PQexec calls with logic like this:
>
> // Submit the query
> if (! PQsendQuery(conn, query))
> reportTheError();
> // Wait for and process result(s)
> while ((result = PQgetResult(conn)) != NULL) {
> switch (PQresultStatus(result)) {
> ... process result, for example:
> case PGRES_COPY_IN:
> // ... copy data here ...
> if (PQendcopy(conn))
> reportTheError();
> break;
> ...
> }
> PQclear(result);
> }
> // When fall out of loop, we're done and ready for a new query
>
> Note that PQgetResult will always report errors by returning a PGresult
> with status PGRES_NONFATAL_ERROR or PGRES_FATAL_ERROR, not by returning
> NULL (since NULL implies non-error termination of the processing loop).
>
> PQexec() will be implemented as follows:
>
> if (! PQsendQuery(conn, query))
> return makeEmptyPGresult(conn, PGRES_FATAL_ERROR);
> lastResult = NULL;
> while ((result = PQgetResult(conn)) != NULL) {
> PQclear(lastResult);
> lastResult = result;
> }
> return lastResult;
>
> This maintains the current behavior that the last result of a series
> of commands is returned by PQexec. (The old implementation is only
> capable of doing that correctly in a limited set of cases, but in the
> cases where it behaves usefully at all, that's how it behaves.)
>
> There is a small difference in behavior, which is that PQexec will now
> return a PGresult with status PGRES_FATAL_ERROR in cases where the old
> implementation would just have returned NULL (and set conn->errorMessage).
> However, any correctly coded application should handle this the same way.
>
> In the above examples, the frontend application is still synchronous: it
> blocks while waiting for the backend to reply to a query. This is often
> undesirable, since the application may have other work to do, such as
> responding to user input. Applications can now handle that by using
> PQisBusy and PQconsumeInput along with PQsendQuery and PQgetResult.
>
> The general idea is that the application's main loop will use select()
> to wait for input (from either the backend or its other input sources).
> When select() indicates that input is pending from the backend, the app
> will call PQconsumeInput, followed by checking PQisBusy and/or PQnotifies
> to see what has happened. If PQisBusy returns FALSE then PQgetResult
> can safely be called to obtain and process a result without blocking.
>
> Note also that NOTIFY messages can arrive asynchronously from the backend.
> They can be detected *without issuing a query* by calling PQconsumeInput
> followed by PQnotifies. I expect a lot of people will build "partially
> async" applications that detect notifies this way but still do all their
> queries through PQexec (or better, PQsendQuery followed by a synchronous
> PQgetResult loop). This compromise allows notifies to be detected without
> wasting time by issuing null queries, yet the basic logic of issuing a
> series of queries remains simple.
>
> Finally, since the application can retain control while waiting for a
> query response, it becomes meaningful to try to cancel a query in progress.
> This is done by calling PQrequestCancel(). Note that PQrequestCancel()
> may not have any effect --- if there is no query in progress, or if the
> backend has already finished the query, then it *will* have no effect.
> The application must continue to follow the result-reading protocol after
> issuing a cancel request. If the cancel is successful, its effect will be
> to cause the current query to fail and return an error message.
>
>
> PROTOCOL CHANGES:
>
> We should change the protocol version number to 2.0.
> It would be possible for the backend to continue to support 1.0 clients,
> if you think it's worth the trouble to do so.
>
> 1. New message type:
>
> Command Done
> Byte1('Z')
>
> The backend will emit this message at completion of processing of every
> command string, just before it resumes waiting for frontend input.
> This change eliminates libpq's current hack of issuing empty queries to
> see whether the backend is done. Note that 'Z' must be emitted after
> *every* query or function invocation, no matter how it terminated.
>
> 2. The RowDescription ('T') message is extended by adding a new value
> for each field. Just after the type-size value, there will now be
> an int16 "atttypmod" value. (Would someone provide text specifying
> exactly what this value means?) libpq will store this value in
> a new "adtmod" field of PGresAttDesc structs.
>
> 3. The "Start Copy In" response message is changed from 'D' to 'G',
> and the "Start Copy Out" response message is changed from 'B' to 'H'.
> These changes eliminate potential confusion with the data row messages,
> which also have message codes 'D' and 'B'.
>
> 4. The frontend may request cancellation of the current query by sending
> a single byte of OOB (out-of-band) data. The contents of the data byte
> are irrelevant, since the cancellation will be triggered by the associated
> signal and not by the data itself. (But we should probably specify that
> the byte be zero, in case we later think of a reason to have different
> kinds of OOB messages.) There is no specific reply to this message.
> If the backend does cancel a query, the query terminates with an ordinary
> error message indicating that the query was cancelled.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> Tom, just wondering were we are with this. Can you update libpq.3?
Will do. I've been busy with other stuff and have not had time to spend
on pgsql, but will try to push it up the priority list.
> Are the sgml sources updated with the protocol changes?
Yes, those are done. I had been hoping not to have to update libpq.3
manually, that's all...
> Also, are these items completed? How about our cancel query key? I
> think it is random/secure enough for our purposes. Can you make the
> changes, or do you need changes from me?
I haven't looked to see what's been done there --- have you finished an
initial implementation, or is the cancel-via-postmaster-instead-of-OOB
change still just at the talk stage? I can handle the libpq end of it,
but feel less secure about modifying the postmaster.
regards, tom lane
> Bruce Momjian <maillist@candle.pha.pa.us> writes: > > Tom, just wondering were we are with this. Can you update libpq.3? > > Will do. I've been busy with other stuff and have not had time to spend > on pgsql, but will try to push it up the priority list. > > > Are the sgml sources updated with the protocol changes? > > Yes, those are done. I had been hoping not to have to update libpq.3 > manually, that's all... > > > Also, are these items completed? How about our cancel query key? I > > think it is random/secure enough for our purposes. Can you make the > > changes, or do you need changes from me? > > I haven't looked to see what's been done there --- have you finished an > initial implementation, or is the cancel-via-postmaster-instead-of-OOB > change still just at the talk stage? I can handle the libpq end of it, > but feel less secure about modifying the postmaster. Perhaps you can work up a patch, and I can review it, or give ideas on how to proceed. The actual protocol layers in the postmaster, especially for authentication, are very hard for me to understand. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)