Re: Working on huge RAM based datasets

>The disk cache on most operating systems is optimized.  Plus, keeping
shared buffers low gives you more room to bump up the sort memory, which
will make your big queries run faster.

Thanks merlin,

Whether the OS caches the data or PG does, you still want it cached. If your
sorting backends gobble up the pages that otherwise would be filled with the
database buffers, then your postmaster will crawl, as it'll *really* have to
wait for stuff from disk. In my scenario, you'd spec the machine so that
there would be plenty of memory for *everything*.

On your OS optimisation point, OS caches are, of course, optimised. But
people have told me that PG's caching strategy is simply less well
optimised, and *that* is the reason for keeping the shared buffer cache down
in my scenario. That's a shame in a way, but I understand why it is the way
it is - other things have been addressed which speed up operations in
different ways. My 'all in RAM' scenario is very rare at the moment, so why
waste valuable development resources on developing optimised RAM based data
structures to hold the data for quicker query execution when hardly anyone
will see the benefit?

However - it won't be so rare for too much longer... If I gave you a half a
terabyte of RAM and a 4 processor 64 bit machine, I'm sure you could imagine
how much quicker databases could run if they were optimised for this sort of
platform.

Anyway, I'm looking forward to experimenting with stuff the way it works at
the moment.

Many thanks,
Andy