Re: libpq support for arrays and composites

On 6/8/08, Andrew Dunstan <andrew@dunslane.net> wrote:
> Tom Lane wrote:
> > Andrew Dunstan <andrew@dunslane.net> writes:
> > > One complicating factor I see is that there is no protocol level support
> for anything other than simple objects - each data value is simply a stream
> of bytes of a known length. We would therefore need some pretty robust
> processing to pick apart structured objects.
> >
> > Are you intending that these operations support both text and binary
> > results?
>
> I'm a bit open on that.

IMO, support for binary is critical.  Because of the interplay of the
array and composite out formats, the number of backslashes grows
exponentially (!) with nesting levels.  This makes text format arrays
unsuitable for any non-trivial operations involving arrays of
composites.

> > The array accessors with ... parameter lists strike me as a bit
> > dangerous, because there is no way at all to verify that the caller is
> > passing the expected number of dimensions.  Can't that be made tighter?
>
> Well, the only alternative I can think of is to make the client walk the
> array one dimension at a time. Something like:
>
> PQarray * PQgetInnerArray(PQarray * array, int dim);

Dimension walking is probably the best approach.  The pure varargs
version which Andrew D is suggesting has too many arguments based
strictly on convention IMO (meaning, I agree with Tom).

One alternative is to do a MAXDIM (6) argument 'getter' also taking
the requested dimension with perhaps some wrapping macros for
simplicity.  One issue with this is that it seems to suggest array
slicing etc. which seems more complicated than it's worth.

One small style note here: the structs in libpq generally have a 'PG'
prefix...PGarray, etc.

> then when we're down to the leaf level, we could have:
>
> int PQgetArrayElementLength(PQarray * array, int dim);
> bool PQgetArrayElementIsNull(PQarray * array, int dim);
> char * PQgetArrayElement(PQarray * array, int dim);
>
> That strikes me as somewhat more cumbersome, so I guess the question is
> whether it's worth it.  It probably fits the slightly clunky feel of libpq.

Have you considered using a PGresult to present the array/composite?
The idea is to arrange it so: one column per composite field and one
row per array row.  I think this is a pretty elegant fit over the
requirements of getting information out of an array, and you have to
add a lot less convetions to libpq.   This is a semantic enhancement
of some of the existing functions but

Tom Lane wrote:
> It might also be useful to provide some functions that form an array or
> composite value from per-element strings, ie, the converse of the
> de-construction routines.  Here I'd be happy to skip the binary case.

That seems a little inconsistent to me: (we allow sending binary query
parameters except when sending arrays).  ISTM that arrays are one of
the cases where sending in binary could be a big win in terms of
performance.

Andrew Dunstan wrote:
> Yeah, that had occurred to me. Will think about it more, although it could
> possibly be done as a separate project, too.

I can't help but notice you proposing some overlapping behavior with
libpqtypes when we could be making better progress working together
and extending libpq in a way that:
*) doesn't introduce new behavior to libpq which is an overreach of
the library (particularly things which can be achieved via simple
wrapping strategies)
*) provides a reasonable foundation for the objectives which
libpqtypes is trying to achieve.

I have always thought that array and composite presentation belongs in
libpq.  For example, since the 'inner result' is constructed on the
libpqtype side, the result has to be built with a deep copy.  This is
fine in most cases but a potential optimzation target if everything is
done in libpq.  Also we could drop some portions of our hooking patch,
namely PQsetValue, PQcopyResult, and possibly PQresultAlloc.

While we didn't introduce text/composite array parsing into
libpqtypes, there is no reason why we couldn't (certain functions are
undefined

I understand that some portion of libpqtypes are unlikely to make it
into libpq.  Is it feasible come up with an array composite approach
that is useful from libpqtypes perspective so we can layer on top of
what andrew d is proposing.

For example, in libpq:
*) array/composite get/put (in text/binary)
*) 'PGparam' argument stacker building arrays/composites for
*) introduce PGarray struct.  If PGresult is used to present data,
AIUI the 'composite' family of functions, outside of what is needed to
present the result from the composite, does not need to exist.

But not:
*) type serializing/deserializing from binary (except in the special
case of arrays/composites).
*) printf/scanf style query functions.

Which we would presumably keep in the libpqtypes library.  I haven't
thought it completely through, but I'm guessing it's possible to
adjust the libpqtype library so that it gives the interface we like
but allow extending libpq to naturally allow better handling of
arrays/composites which we need and libpq users obviously want.

Our propsosed libpqtypes library solves all the use cases regarding
arrays/composites with the exception of pulling out arrays in text
(but it could be easily adjusted to do that, too).  So, maybe it's
worthwhile seeing if we can adapt some of the work which we have
already done and is working quite well.

merlin