On 04/23/2015 04:04 PM, Andrew Gierth wrote:
>>>>>> "Joshua" == Joshua D Drake <jd@commandprompt.com> writes:
> Joshua> The database dumps fine as long as we don't dump large
> Joshua> objects. However, if we try to dump the large objects, FreeBSD
> Joshua> will kill pg_dump as it will consume all free memory and
> Joshua> swap. With Andrew's help we were able to determine the
> Joshua> following:
>
> Joshua> There is a memory cost of about 160 bytes per largeobject.
>
> I may have the exact number here wrong, it was just a quick eyeball of
> the data structures (and depends on malloc overheads anyway).
>
> The relevant code is getBlobs in pg_dump.c, which queries the whole of
> pg_largeobject_metadata without using a cursor (so the PGresult is
> already huge thanks to having >100 million rows), and then mallocs a
> BlobInfo array and populates it from the PGresult, also using pg_strdup
> for the oid string, owner name, and ACL if any.
>
I'm surprised this hasn't come up before. I have a client that I
persuaded to convert all their LOs to bytea fields because of problems
with pg_dump handling millions of LOs, and kept them on an older
postgres version until they made that change.
cheers
andrew