COPY with fk's slow

I'm playing around with importing 10,000 records into a table with a single index and a single foreign key. COPY takes roughly 10 minutes(!). However, if I drop the foreign key, COPY takes a couple seconds, THEN adding the foreign key back takes another few seconds. Any idea why COPY with a f/k takes so much longer? Any workarounds? Such as lock the table, drop fk, copy, add fk, unlock?

John