Re: exposing COPY API

On 02/07/2011 11:34 AM, Andrew Dunstan wrote:
>
>
> On 02/04/2011 05:49 AM, Itagaki Takahiro wrote:
>> Here is a demonstration to support jagged input files. It's a patch
>> on the latest patch. The new added API is:
>>
>>    bool NextLineCopyFrom(
>>          [IN] CopyState cstate,
>>          [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid)
>>
>> It just returns separated fields in the next line. Fortunately, I need
>> no extra code for it because it is just extracted from NextCopyFrom().
>>
>> I'm willing to include the change into copy APIs,
>> but we still have a few issues. See below.
>
>
> I've looked at this, and I think it will do what I want. I haven't had
> time to play with it, although I hope to soon.  AIUI, it basically
> hands back the raw parsed strings to the user, who then has the
> responsibility of constructing Datum and Nulls arrays to form the
> tuple.  That should be all I need. So +1 from me for including it. In
> fact, +10. And many thanks.
>
>
> I think we need a better name though. NextCopyFromRawFields maybe.


Here is a patch against the latest revision of file_fdw to exercise this
API. It includes some regression tests, and I think apart from one or
two small details plus a requirement for documentation, is complete.

This work is also published at
<https://github.com/adunstan/postgresql-dev/tree/sqlmed3>

Here's an excerpt from the regression tests:

    CREATE FOREIGN TABLE jagged_text (
         t   text[]
    ) SERVER file_server
    OPTIONS (format 'csv', filename
    '/home/andrew/pgl/pg_head/contrib/file_fdw/data/jagged.csv', header
    'true', textarray 'true');
    SELECT * FROM jagged_text;
                          t
    --------------------------------------------
      {"one field"}
      {two,fields}
      {three,NULL,"fields of which one is null"}
      {"next line has no fields"}
      {}
    (5 rows)

    SELECT t[3] AS a, t[1] AS b, t[99] AS c  FROM jagged_text;
                   a              |            b            | c
    -----------------------------+-------------------------+---
                                  | one field               |
                                  | two                     |
      fields of which one is null | three                   |
                                  | next line has no fields |
                                  |                         |
    (5 rows)

Overall, this API works quite nicely, and does exactly what I want, so
again many thanks.

cheers

andrew