> imo, the key to high performance big data movements in postgresql is
> mastering sql and pl/pgsql, especially the latter. once you get good
> at it, your net time of copy+plpgsql is going to be less than
> insert+tcl.
If this implies bulk inserts, I'm afraid I have to consider something else.
Any data that has been imported and dedpulicated has to be placed back into
the database so that it can be available for the next imported row (there
are currently 16 tables affected, and more to come). If I was to cache all
inserts into a seperate resource, then I would have to search 32 tables -
the local pending resources, as well as the data still in the system. I am
not even mentioning that imports do not just insert rows, they could just
rows, adding their own complexity.