Re: Streaming large data into postgres [WORM like applications]

At 8:49p on 12 May 2007, Dhaval Shah wrote:
> That leads to the question, can the data be compressed? Since the data
> is very similar, any compression would result in some 6x-10x
> compression. Is there a way to identify which partitions are in which
> data files and compress them until they are actually read?

There was a very interesting article in ;login: magazine in April of
this year discussing how they dealt with an exorbitant amount of largely
similar data.  The article claimed that through aggregation and gzip
compression, they were able to reduce what they needed to store by
roughly 350x, or about .7 bytes per 'event'.  The article is

The Secret Lives of Computers Exposed: Flight Data Recorder for Windows
by Chad Verbowski

You might try to get your mitts on that article for some ideas.  I'm not
sure you could apply any of their ideas directly to the Postgres backend
data files, but perhaps somewhere in your pipeline.

Kevin