Here is a method of filtering out the duplicate keys from the input file:
Let us assume that the input data is in /tmp/table.in, that that file is
tab-delimited and that the primary key is in field 2.
psql -d database -c "COPY table TO '/tmp/table.1'" psql -d database -c "COPY table TO '/tmp/table.2'" cat
/tmp/table.in/tmp/table.[12] | sort -k 2 -t \[tab] | uniq -u -W 1 -f 1 -t \[tab] >/tmp/table.in.unique
[tab] stands for the sequence "ctrl-V tab", which will force an actual tab
character into the command line (the backslash will protect it from the
shell).
We copy the database out twice to ensure that all keys already in it are
excluded, otherwise we would be reintroducing all primary keys that were
_not_ in the desired input.
The end product is a file that excludes all primary keys that are already
in the target table.
--
Oliver Elphick Oliver.Elphick@lfix.co.uk
Isle of Wight http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47 6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C
======================================== "Is any one of you in trouble? He should pray. Is anyone happy? Let him
singsongs of praise. Is any one of you sick? He should call the elders of the church to pray over him...The
prayerof a righteous man is powerful and effective." James 5:13,14,16