On 11/21/23 13:59, Andres Freund wrote:
>
> On 2023-11-21 13:41:15 -0400, David Steele wrote:
>> On 11/20/23 16:41, Andres Freund wrote:
>>>
>>> On 2023-11-20 15:56:19 -0400, David Steele wrote:
>>>> I understand this is an option -- but does it need to be? What is the
>>>> benefit of excluding the manifest?
>>>
>>> It's not free to create the manifest, particularly if checksums are enabled.
>>
>> It's virtually free, even with the basic CRCs.
>
> Huh?
<snip>
> I'd not call 7.06->4.77 or 6.76->4.77 "virtually free".
OK, but how does that look with compression -- to a remote location?
Uncompressed backup to local storage doesn't seem very realistic. With
gzip compression we measure SHA1 checksums at about 5% of total CPU.
Obviously that goes up with zstd or lz4. but parallelism helps offset
that cost, at least in clock time.
I can't understate how valuable checksums are in finding corruption,
especially in long-lived backups.
>> Anyway, would you really want a backup without a manifest? How would you
>> know something is missing? In particular, for page incremental how do you
>> know something is new (but not WAL logged) if there is no manifest? Is the
>> plan to just recopy anything not WAL logged with each incremental?
>
> Shrug. If you just want to create a new standby by copying the primary, I
> don't think creating and then validating the manifest buys you much. Long term
> backups are a different story, particularly if data files are stored
> individually, rather than in a single checksummed file.
Fine, but you are probably not using page incremental if just using
pg_basebackup to create a standby. With page incremental, at least one
of the backups will already exist, which argues for a manifest.
>>> Also, for external backups, there's no manifest...
>>
>> There certainly is a manifest for many external backup solutions. Not having
>> a manifest is just running with scissors, backup-wise.
>
> You mean that you have an external solution gin up a backup manifest? I fail
> to see how that's relevant here?
Just saying that for external backups there *is* often a manifest and it
is a good thing to have.
Regards,
-David