Re: COPY enhancements

On Thu, 2009-10-08 at 11:01 -0400, Tom Lane wrote:

> So as far as I can see, the only form of COPY error handling that
> wouldn't be a cruel joke is to run a separate subtransaction for each
> row, and roll back the subtransaction on error.  Of course the
> problems
> with that are (a) speed, (b) the 2^32 limit on command counter IDs
> would mean a max of 2^32 rows per COPY, which is uncomfortably small
> these days.  Previous discussions of the problem have mentioned trying
> to batch multiple rows per subtransaction to alleviate both issues.
> Not easy of course, but that's why it's not been done yet.  With a
> patch like this you'd also have (c) how to avoid rolling back the
> insertions into the logging table.

(d) using too many xids will force the system to begin immediate
wraparound-avoidance vacuuming to freeze rows. 

Dimitri's pgloader is looking even more attractive, not least because it
exists and it works. (And is the reason I personally stopped considering
the COPY-error-logging feature as important).

-- Simon Riggs           www.2ndQuadrant.com