Re: Practical impediment to supporting multiple SSL libraries

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > It's only the functional equivalent when you think all the world is a
> > Postgres app, which is just not the case.
>
> If we are dumping data into a simple memory block in a format dictated
> by libpq, then we haven't done a thing to make the app's use of that
> data independent of libpq.  Furthermore, because that format has to be
> generalized (variable-length fields, etc), it will not be noticeably
> easier to use than the existing PQresult API.

The format of the structure *isn't* really dictated by libpq.  The
offsets, value length and record size is intended to support most any
C array structure.  Variable length fields have a max size which, if it
goes over, an error is returned or indicated throgh the indicator
array.  It also gets it into the structure quite a few applications
would like to have it in (which is certainly not PQresult).

> What I would envision as a typical use of a callback is to convert the
> data and store it in a C struct designed specifically for a particular
> query's known result structure (say, a few ints, a string of a known
> maximum length, etc).  libpq can't do that, but a callback could do it
> easily.

Heh, this is exactly what I'm proposed we make libpq capable of doing,
which is a relatively simple thing to do.  I agree that it's often a
goal of application devlopers to get it into this structure.  The one
downside is that at the moment I think the binary results from libpq
come back in network-byte-order instead of host-byte-order.  Oracle
provided a way to indicate the types of the fields in the structure and
performed some conversions (such as these) for you.  The constants they
used started with "SQL_" but I'm not entirely sure if they were actually
defined in the standard or not.

> The fixed-memory-block approach also falls over when considering results
> of uncertain maximum size.  Lastly, it doesn't seem to me to respond at
> all to the ODBC needs that started this thread: IIUC, they want each row
> separately malloc'd so that they can free selected rows from the
> completed resultset.

Results of uncertain maximum size aren't a problem at all...  The caller
can do the exact same thing libpq does (realloc), or it could allocate
another array.  *Each* call to the libpq function would return the
number of elements actually populated into the memory-block; the caller
would then be expected to pass in a *fresh* memory block for the next
call (which could just be a simply calculated offset into the block they
allocated, or could be a realloc'd block + offset, or a brand new block,
etc...).

I'm really not why there seem to be this "this won't work!" reaction.
This isn't something I came up with out of whole cloth, it's an API that
isn't unlike PQexecParams, is similar to something Oracle does (which
I've used quite a bit for doing *exactly* what's mentioned above- I've
got an array of pre-defined C structs that I know match the query and
I want that array filled in) and is really not that complicated.

> > For one thing, it's certainly possible the callback (to do a data
> > transform like you're suggesting) would want access to the other
> > information in a given tuple.  Having to store a partial tuple in a
> > temporary area which has to be built up to the full tuple before you can
> > actually process it wouldn't be all that great.
>
> So instead, you'd prefer to *always* store partial tuples in a temporary
> area, thereby making sure the independent-field-conversions case has
> performance just as bad as the dependent-conversions case.
> I can't follow that reasoning.

I havn't been ruling out providing a callback mechanism as well but I
think it's the 10% case and the 90% case is being shoe-horned into the
10% case with a performance degredation to boot.  They're also not
partial tuples, it's not a temporary area, and there's demonstratably
less copying around of the data.  It seems ODBC may be in the 10% piece
here but I havn't looked at the ODBC source code yet.
Thanks,
    Stephen