* John R Pierce (pierce@hogranch.com) wrote:
> if we assume the tables average 1KB/record (which is a fairly large
> record size even including indexing), you're looking at 400 billion
> records. if you can populate these at 5000 records/second, it
> would take 2.5 years of 24/7 operation to populate that.
5000 1KB records per second is only 5MB/s or so, which is really quite
slow.. I can't imagine that they'd load all of this data by doing a
commit for each record and you could load a *huge* amount of data *very*
quickly, in parallel, by using either unlogged tables or wal_level =
minimal and creating the tables in the same transaction that's loading
them.
> this sort of big data system is probably more suitable for something
> like hadoop+mongo or whatever on a cloud of 1000 nodes, not a
> monolithic SQL relational database.
Or a federated PG database using FDWs..
Sadly, I've not personally worked with a data system on the 100+TB range
w/ PG (we do have a Hadoop environment along that scale) but I've built
systems as large as 25TB which, built correctly, work very well. Still,
I don't think I'd recommend building a single-image PG database on that
scale but rather would shard it.
Thanks,
Stephen