On 02/04/2011 05:49 AM, Itagaki Takahiro wrote:
> Here is a demonstration to support jagged input files. It's a patch
> on the latest patch. The new added API is:
>
> bool NextLineCopyFrom(
> [IN] CopyState cstate,
> [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid)
>
> It just returns separated fields in the next line. Fortunately, I need
> no extra code for it because it is just extracted from NextCopyFrom().
>
> I'm willing to include the change into copy APIs,
> but we still have a few issues. See below.
I've looked at this, and I think it will do what I want. I haven't had
time to play with it, although I hope to soon. AIUI, it basically hands
back the raw parsed strings to the user, who then has the responsibility
of constructing Datum and Nulls arrays to form the tuple. That should
be all I need. So +1 from me for including it. In fact, +10. And many
thanks.
I think we need a better name though. NextCopyFromRawFields maybe.
> On Fri, Feb 4, 2011 at 16:53, Andrew Dunstan<andrew@dunslane.net> wrote:
>> The problem with COPY FROM is that nobody's come up with a good syntax for
>> allowing it as a FROM target. Doing what I want via FDW neatly gets us
>> around that problem. But I'm quite OK with doing the hard work inside the
>> COPY code - that's what my working prototype does in fact.
> I think it is not only syntax issue. I found an issue that we hard to
> support FORCE_NOT_NULL option for extra fields. See FIXME in the patch.
> It is a fundamental problem to support jagged fields.
I don't think we need to worry about it. The caller will have access to
the raw strings so they can handle it. In fact, I'd take out that bit of
code in NextCopyLine_From, and replace it with a comment about how it's
the caller's responsibility to handle.
>> One thing I'd like is to to have file_fdw do something we can't do another
>> way. currently it doesn't, so it's nice but uninteresting.
> BTW, how do you determine which field is shifted in your broken CSV file?
> For example, the case you find "AB,CD,EF" for 2 columns tables.
> I could provide a raw CSV reader for jagged files, but you still have to
> cook the returned fields into a proper tuple...
>
I answered this previously, but in the case of a text array it won't
even arise - the array will have however many fields have been read.
cheers
andrew