Just wanted to say thanks for your help with my performance question. You have given me plenty of things to investigate. Further, I think the problem is almost certainly with my app, so I need to do more work there!
I really like the idea of just loading everything in memory and then dumping it all out later. I have 6 GB RAM, so it should be plenty to handle this.