Hi Robert,
On 11/08/2010 10:23 PM, Robert Haas wrote:
> On Sat, Oct 30, 2010 at 9:30 PM, Arturas Mazeika<mazeika@gmail.com> wrote:
>
>> Thanks for the info, this explains a lot.
>>
>> Yes, I am upgrading from the 32bit version to the 64bit one.
>>
>> We have pretty large databases (some over 1 trillion of rows, and some
>> containing large documents in blobs.) Giving a bit more memory than 4GB
>> limit to Postgres was what we were long longing for. Postgres was able to
>> handle large datasets (I suppose it uses something like long long (64bit)
>> data type in C++) and I hoped naively that Postgres would be able to migrate
>> from one version to the other without too much trouble.
>>
>> I tried to pg_dump one of the DBs with large documents. I failed with out of
>> memory error. I suppose it is rather hard to migrate in my case :-( Any
>> suggestions?
>>
> Yikes, that's not good. How many tables do you have in your database?
> How many large objects? Any chance you can coax a stack trace out of
> pg_dump?
>
We are storing the UKGOV Web archive [1] in a Postgres DB. There are two
large tables and a dozen of small tables. The large tables are pages and
links. Pages describe information about the Web page, including its
compressed content (this is usually tens KBs, sometimes MBs). Links
describe link structure between the Web pages. The pages table occupies
some 176GB (50M rows), while links occupy 32GB (500M rows). All
databases that need to be migrated occupy 1.07TB.
BTW, in Postgres, the shared buffers is set to 128M, and the working mem
is set to 1GB. We've got 16GB memory in total (the machine is rather an
older one).
I'll try to dump the DB again, and get the stack trace. This may take
quite a while.
[1] http://www.europarchive.org/ukgov.php