Hi all,
on a daily basis, I load records from an external source into my database. During the load, the records go through a validation process during which I have to read anywhere between 2 and 15 records from the database.
The postmaster process which does this, starts out with 10 MB memory usage but soon swells up to about 500 MB (on a normal day). Some days, I receive a larger amount of data and the process swells up to about 1.4 GB before it bombs out with a 'memory exhausted' message.
I believe, the answer is to have the transactions smaller, but since PL/PGSQL runs everything is one big transaction, how would I go about that? Can I temporarily disable transaction processing all together for this process? (if the load fails, I can easily delete the created records by hand).
Best regards,
Chris