Folks,
I am currently testing a massive data update function, which takes a
large amount of raw data (in 3 tables comprising over 2 million records
total) and organizes and cleans that data in a series of successive
steps for the final relational database.
The problem is, it doesn't complete. About 1/3 of the way in to the
data updates, postgres gets stuck in an endless cycle of recycling
transaction logs and child processes and won't finish (current longest
abort time is 37 minutes).
I've upped the various memory parameters significantly ... working with
wal_files 32, wal_buffers 32, sort_mem 65536, shared_buffers 256, etc.The CPU (1400mhz athalon) and memory (512Mb DDR
RAM)are hardly being
taxed. The only hardware bottleneck I can see is that the system only
has one IDE drive (though a fast, large one). I've added many indexes.
Does anyone have an suggestions on making this process complete, short
of running out and buying a RAID SCSI array?
-Josh Berkus