Re: Confusing error message with too-large file in pg_basebackup

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Guillaume Lelarge wrote:
>> Looking at the file's size is probably a better idea. As far as I know,
>> PostgreSQL doesn't create files bigger than 1GB, except for log files. I'm
>> not sure about this but I guess pg_basebackup doesn't ship log files. So,
>> looking at the size would work.

> Hmm, so we let configure --with-segsize to change the file size.  The
> configure help says that the limit should be "less than your OS' limit
> on file size".  We don't warn them that this could cause backup
> problems later on.  Should we add a blurb about that somewhere?

Actually ... why don't we get rid of the limit?  wikipedia's entry on
tar format says

    ... only 11 octal digits can be stored. This gives a maximum file size
    of 8 gigabytes on archived files. To overcome this limitation, star in
    2001 introduced a base-256 coding that is indicated by setting the
    high-order bit of the leftmost byte of a numeric field. GNU-tar and
    BSD-tar followed this idea.

If that extension is as widespread as this suggests, then following it
when we have a file > 8GB seems like a better answer than failing
entirely.  If you try to read the dump with an old tar program, old
pg_restore, etc, it might fail ... but are you really worse off than
if you couldn't make the dump at all?

            regards, tom lane