Re: backup manifests

Hi,

On 2020-03-30 15:23:08 -0400, Robert Haas wrote:
> On Mon, Mar 30, 2020 at 2:59 PM Andres Freund  wrote:
> > I wonder if it'd not be best, independent of whether we build in this
> > verification, to include that metadata in the manifest file. That's for
> > sure better than having to build a separate tool to parse timeline
> > history files.
> 
> I don't think that's better, or at least not "for sure better". The
> backup_label going to include the START TIMELINE, and if -Xfetch is
> used, we're also going to have all the timeline history files. If the
> backup manifest includes those same pieces of information, then we've
> got two sources of truth: one copy in the files the server's actually
> going to read, and another copy in the backup_manifest which we're
> going to potentially use for validation but ignore at runtime. That
> seems not great.

The data in the backup label isn't sufficient though. Without having
parsed the timeline file there's no way to verify that the correct WAL
is present. I guess we can also add client side tools to parse
timelines, add command the fetch all of the required files, and then
interpret that somehow.

But that seems much more complicated.

Imo it makes sense to want to be able verify that WAL looks correct even
transporting WAL using another method (say archiving) and thus using
pg_basebackup's -Xnone.

For the manifest to actually list what's required for the base backup
doesn't seem redundant to me. Imo it makes the manifest file make a good
bit more sense, since afterwards it actually describes the whole base
backup.

Taking the redundancy agreement a bit further you can argue that we
don't need a list of relation files at all, since they're in the catalog
:P. Obviously going to that extreme doesn't make all that much
sense... But I do think it's a second source of truth that's independent
of what the backends actually are going to read.

Greetings,

Andres Freund