Re: pg_basebackup failed to back up large file

Andres Freund <andres@2ndquadrant.com> writes:
> On 2014-06-03 11:04:58 -0400, Tom Lane wrote:
>> My point is that having backups crash on an overflow doesn't really seem
>> acceptable.  IMO we need to reconsider the basebackup protocol and make
>> sure we don't *need* to put values over 4GB into this field.  Where's the
>> requirement coming from anyway --- surely all files in PGDATA ought to be
>> 1GB max?

> Fujii's example was logfiles in pg_log. But we allow to change the
> segment size via a configure flag, so we should support that or remove
> the ability to change the segment size...

What we had better do, IMO, is fix things so that we don't have a filesize
limit in the basebackup format.  After a bit of googling, I found out that
recent POSIX specs for tar format include "extended headers" that among
other things support member files of unlimited size [1].  Rather than
fooling with partial fixes, we should make the basebackup logic use an
extended header when the file size is over INT_MAX.
        regards, tom lane

[1] http://pubs.opengroup.org/onlinepubs/9699919799/
see "pax" under shells & utilities