Обсуждение: some more rambling on the new fe/be communication
it would appear that libpq now has a outgoing data buffer associated with PGconn struct which only gets sent (with send()!?) when pqFlush gets called. the backend still appears to use and pass FILE * for reading and writing. I wasn't aware that you can read data from a FILE * sent with send() over a socket. Is this portable? Time to pull out stevens. In any case, I don't think this bodes well for my SSL patch -- and I think I've missed something -- why have we switched to send/recv? I assume for the synchronous notification? I haven't been following that discussion as much as I possibly could be, so I'll look in the archives. Anyway, this is sort of a plea for help -- I'm totally confused, so if there's just something I'm missing, please let me know. Does anyone know what implications the new communication scheme has for SSL? I know this isn't a postgresql priority, but it is an interest of mine. Is it still possible? I'll start doing my homework.
Brett McCormick <brett@work.chicken.org> writes:
> [ Brett was unpleasantly surprised to find major changes in libpq ]
Sorry about that, Brett. This was discussed on the hackers list a month
or so ago, but evidently you missed the thread. I made some fairly
major changes in the client-side libpq to allow it to be used
asychronously, that is without blocking until the completion of a query.
I didn't bother to do much cleanup of the backend side, since it didn't
have to change to get the functionality I was after. I agree that it
could stand a cleanup, so if you want to do it, by all means do.
> it would appear that libpq now has a outgoing data buffer associated
> with PGconn struct which only gets sent (with send()!?) when pqFlush
> gets called. the backend still appears to use and pass FILE * for
> reading and writing. I wasn't aware that you can read data from a
> FILE * sent with send() over a socket. Is this portable?
Yes. Data on a connection is data; there's no way for the far end to
tell what syscall or library was used to collect and send the data.
(The far end might not even be Unix or C based, after all.)
> In any case, I don't think this bodes well for my SSL patch -- and I
> think I've missed something -- why have we switched to send/recv?
Because going through the stdio library gives up control over blocking
when no data is available. getc() will block, period.
Does SSL support non-blocking recv? If so it shouldn't be hard to put
an SSL layer under what I did with libpq. Note the existence of
PQsocket() however. If an SSL connection can't be select()'d for then
we have got trouble.
BTW, I believe I did fix your earlier complaint that the backend called
pq_putstr again after closing the client connection.
regards, tom lane
On Thu, 4 June 1998, at 13:23:17, Tom Lane wrote: > Sorry about that, Brett. This was discussed on the hackers list a month > or so ago, but evidently you missed the thread. I made some fairly > major changes in the client-side libpq to allow it to be used > asychronously, that is without blocking until the completion of a query. don't be sorry! that's a good thing. not sure how I missed that. > I didn't bother to do much cleanup of the backend side, since it didn't > have to change to get the functionality I was after. I agree that it > could stand a cleanup, so if you want to do it, by all means do. I beleive I will. I thought about this last night, and I came up with this: Since what we need is secure database connections under a stable release, I'll continue to develop my SSL patch for 6.3.2. Since I've already done the work of "cleaning up" (I use quotes because all I've really done is changed the functions to pass the struct ptr around, and isolated all read/writes to two functions, pq_read & pq_write) I'll issue two separate patches. One which modularizes the IO a little which will make it easy for people who wish to add other layers (like kerberos encryption) and an SSL patch to run on top of that. I'll have to familiarize myself with the new frontend code, but I plan on making a similar patch for 6.4 (as we'll also want SSL connections with that). I am tempted to hold off on that once I get my current SSL up to snuff and instead work on perl stored procedures, as I feel that is more valuable (and will also do more to familiarize myself with the code). > > > it would appear that libpq now has a outgoing data buffer associated > > with PGconn struct which only gets sent (with send()!?) when pqFlush > > gets called. the backend still appears to use and pass FILE * for > > reading and writing. I wasn't aware that you can read data from a > > FILE * sent with send() over a socket. Is this portable? > > Yes. Data on a connection is data; there's no way for the far end to > tell what syscall or library was used to collect and send the data. > (The far end might not even be Unix or C based, after all.) What about OOB data? is that just data as well? > > > In any case, I don't think this bodes well for my SSL patch -- and I > > think I've missed something -- why have we switched to send/recv? > > Because going through the stdio library gives up control over blocking > when no data is available. getc() will block, period. > > Does SSL support non-blocking recv? If so it shouldn't be hard to put > an SSL layer under what I did with libpq. Note the existence of > PQsocket() however. If an SSL connection can't be select()'d for then > we have got trouble. I'm sure an SSL connection can be select()'d, and it does support non-blocking recv (I think that's the only way). I think it does block, however, if it doesn't get a full SSL "packet" (or whatever the appropriate term may be). > BTW, I believe I did fix your earlier complaint that the backend called > pq_putstr again after closing the client connection. excellent.
Brett McCormick <brett@work.chicken.org> writes:
> I'll have to familiarize myself with the new frontend code, but I plan
> on making a similar patch for 6.4 (as we'll also want SSL connections
> with that).
This seems like a reasonable plan, if you need SSL *now* and not after
6.4 is released. But:
> I am tempted to hold off on that once I get my current
> SSL up to snuff and instead work on perl stored procedures, as I feel
> that is more valuable (and will also do more to familiarize myself
> with the code).
I'd recommend you do the 6.4 version of the patch first, while it's
still fresh in your mind. AFAIK, stored procedures are a completely
different area of the system; you won't learn anything there that is
relevant to the FE/BE protocol.
> On Thu, 4 June 1998, at 13:23:17, Tom Lane wrote:
>> Yes. Data on a connection is data; there's no way for the far end to
>> tell what syscall or library was used to collect and send the data.
>> (The far end might not even be Unix or C based, after all.)
> What about OOB data? is that just data as well?
As far as the TCP protocol is concerned, yes. A lot of libraries that
you might want to use do not have an API that accounts for the separate
"OOB" channel within one TCP connection, so it may be difficult or
impossible to get at TCP's OOB facility from within a particular
programming environment. But that's not exactly a cross-environment
compatibility problem, it's just a missing feature in a library API.
Any two implementations that both handle OOB should be able to
communicate.
> I'm sure an SSL connection can be select()'d, and it does support
> non-blocking recv (I think that's the only way). I think it does
> block, however, if it doesn't get a full SSL "packet" (or whatever the
> appropriate term may be).
Hmm ... I don't know enough about SSL to know if this is really a
problem, but your comment raises warning flags in my head. This
needs investigation.
regards, tom lane
On Thu, 4 June 1998, at 19:10:09, Tom Lane wrote: > > I am tempted to hold off on that once I get my current > > SSL up to snuff and instead work on perl stored procedures, as I feel > > that is more valuable (and will also do more to familiarize myself > > with the code). > > I'd recommend you do the 6.4 version of the patch first, while it's > still fresh in your mind. AFAIK, stored procedures are a completely > different area of the system; you won't learn anything there that is > relevant to the FE/BE protocol. I know -- I'm looking to learn more about other areas of the system. And because the perl stored procedures will support lots of cool backend functions (what sort of stuff is permissible to interface to?) i'll learn about them that way. I've learned as much about the fe/be protocol as I wish to know ;) But I may take your advice. > > I'm sure an SSL connection can be select()'d, and it does support > > non-blocking recv (I think that's the only way). I think it does > > block, however, if it doesn't get a full SSL "packet" (or whatever the > > appropriate term may be). > > Hmm ... I don't know enough about SSL to know if this is really a > problem, but your comment raises warning flags in my head. This > needs investigation. Unfortunately, SSL documention is rather incomplete. thanks for your info, comments and suggestions
> As far as the TCP protocol is concerned, yes. A lot of libraries that > you might want to use do not have an API that accounts for the separate > "OOB" channel within one TCP connection, so it may be difficult or > impossible to get at TCP's OOB facility from within a particular > programming environment. But that's not exactly a cross-environment > compatibility problem, it's just a missing feature in a library API. > Any two implementations that both handle OOB should be able to > communicate. Looks like we will be removing OOB in favor of a CANCEL cookie sent to the postmaster. I will work up something soon. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)