The error you are seeing is triggered because this relation file exceeds MAX_TAR_MEMBER_FILELEN or 8GB for a single tar member, which is as well the norm for tar.
I thought PostgreSQL would break the file if it grows beyond 1GB (1GB is segment size which one can modify while compiling). Am I missing something or have I got it totally wrong?
No, you are missing nothing. Thanks for reminding :) (This is settable with ./configure --with-segsize) So the build has been done with some uncommon option.