Re: block-level incremental backup

On Mon, Apr 15, 2019 at 9:01 AM Stephen Frost <sfrost@snowman.net> wrote:
> I love the general idea of having additional facilities in core to
> support block-level incremental backups.  I've long been unhappy that
> any such approach ends up being limited to a subset of the files which
> need to be included in the backup, meaning the rest of the files have to
> be backed up in their entirety.  I don't think we have to solve for that
> as part of this, but I'd like to see a discussion for how to deal with
> the other files which are being backed up to avoid needing to just
> wholesale copy them.

Ideas?  Generally, I don't think that anything other than the main
forks of relations are worth worrying about, because the files are too
small to really matter.  Even if they're big, the main forks of
relations will be much bigger.  I think.

> I'm quite concerned that trying to graft this on to pg_basebackup
> (which, as you note later, is missing an awful lot of what users expect
> from a real backup solution already- retention handling, parallel
> capabilities, WAL archive management, and many more... but also is just
> not nearly as developed a tool as the external solutions) is going to
> make things unnecessairly difficult when what we really want here is
> better support from core for block-level incremental backup for the
> existing external tools to leverage.
>
> Perhaps there's something here which can be done with pg_basebackup to
> have it work with the block-level approach, but I certainly don't see
> it as a natural next step for it and really does seem like limiting the
> way this is implemented to something that pg_basebackup can easily
> digest might make it less useful for the more developed tools.

I agree that there are a bunch of things that pg_basebackup does not
do, such as backup management.  I think a lot of users do not want
PostgreSQL to do backup management for them.  They have an existing
solution that they use to manage backups, and they want PostgreSQL to
interoperate with it. I think it makes sense for pg_basebackup to be
in charge of taking the backup, and then other tools can either use it
as a building block or use the streaming replication protocol to send
approximately the same commands to the server.  I certainly would not
want to expose server capabilities that let you take an incremental
backup and NOT teach pg_basebackup to use them -- then we'd be in a
situation of saying that PostgreSQL has incremental backup, but you
have to get external tool XYZ to use it.  That will be perceived as
PostgreSQL does NOT have incremental backup and this external tool
adds it.

> As an example, I believe all of the other tools mentioned (at least,
> those that are open source I'm pretty sure all do) support parallel
> backup and therefore having a way to get the block-level changes in a
> parallel fashion would be a pretty big thing that those tools will want
> and pg_basebackup is single-threaded today and this proposal doesn't
> seem to be contemplating changing that, implying that a serial-based
> block-level protocol would be fine but that'd be a pretty awful
> restriction for the other tools.

I mentioned this exact issue in my original email.  I spoke positively
of it.  But I think it is different from what is being proposed here.
We could have parallel backup without incremental backup, and that
would be a good feature.  We could have parallel backup without full
backup, and that would also be a good feature.  We could also have
both, which would be best of all.  I don't see that my proposal throws
up any architectural obstacle to parallelism.  I assume parallel
backup, whether full or incremental, would be implemented by dividing
up the files that need to be sent across the available connections; if
incremental backup exists, each connection then has to decide whether
to send the whole file or only part of it.

> This part of the discussion is a another example of how we're limiting
> ourselves in this implementation to the "pg_basebackup can work with
> this" case- by only consideration the options of "scan all the files" or
> "use the WAL- if the request is for WAL we have available on the
> server."  The other backup solutions mentioned in your initial email,
> and others that weren't, have a WAL archive which includes a lot more
> WAL than just what the primary currently has.  When I've thought about
> how WAL could be used to build a differential or incremental backup, the
> question of "do we have all the WAL we need" hasn't ever been a
> consideration- because the backup tool manages the WAL archive and has
> WAL going back across, most likely, weeks or even months.  Having a tool
> which can essentially "compress" WAL would be fantastic and would be
> able to be leveraged by all of the different backup solutions.

I don't think this is a case of limiting ourselves; I think it's a
case of keeping separate considerations properly separate.  As I said
in my original email, the client doesn't really need to know how the
server is identifying the blocks that have been modified.  That is the
server's job.  I started a separate thread on the WAL-scanning
approach, so we should take that part of the discussion over there.  I
see no reason why the server couldn't be taught to reach back into an
available archive for WAL that it no longer has locally, but that's
really independent of the design ideas being discussed on this thread.

> Two things here- having some file that "stops the server from starting"
> is just going to cause a lot of pain, in my experience.  Users do a lot
> of really rather.... curious things, and then come asking questions
> about them, and removing the file that stopped the server from starting
> is going to quickly become one of those questions on stack overflow that
> people just follow the highest-ranked question for, even though everyone
> who follows this list will know that doing so results in corruption of
> the database.

Wait, you want to make it maximally easy for users to start the server
in a state that is 100% certain to result in a corrupted and unusable
database?  Why?? I'd l like to make that a tiny bit difficult.  If
they really want a corrupted database, they can remove the file.

> An alternative approach in developing this feature would be to have
> pg_basebackup have an option to run against an *existing* backup, with
> the entire point being that the existing backup is updated with these
> incremental changes, instead of having some independent tool which takes
> the result of multiple pg_basebackup runs and then combines them.

That would be really unsafe, because if the tool is interrupted before
it finishes (and fsyncs everything), you no longer have any usable
backup.  It also doesn't lend itself to several of the scenarios I
described in my original email -- like endless incrementals that are
merged into the full backup after some number of days -- a capability
upon which others have already remarked positively.

> An alternative tool might be one which simply reads the WAL and keeps
> track of the FPIs and the updates and then eliminates any duplication
> which exists in the set of WAL provided (that is, multiple FPIs for the
> same page would be merged into one, and only the delta changes to that
> page are preserved, across the entire set of WAL being combined).  Of
> course, that's complicated by having to deal with the other files in the
> database, so it wouldn't really work on its own.

You've jumped back to solving the server's problem (which blocks
should I send?) rather than the client's problem (what does an
incremental backup look like once I've taken it and how do I manage
and restore them?).  It does seem possible to figure out the contents
of modified blocks strictly from looking at the WAL, without any
examination of the current database contents.  However, it also seems
very complicated, because the tool that is figuring out the current
block contents just by looking at the WAL would have to know how to
apply any type of WAL record, not just one that contains an FPI.  And
I really don't want to build a client-side tool that knows how to
apply WAL.

> I'd really prefer that we avoid adding in another low-level tool like
> the one described here.  Users, imv anyway, don't want to deal with
> *more* tools for handling this aspect of backup/recovery.  If we had a
> tool in core today which managed multiples backups, kept track of them,
> and all of the WAL during and between them, then we could add options to
> that tool to do what's being described here in a way that makes sense
> and provides a good interface to users.  I don't know that we're going
> to be able to do that with pg_basebackup when, really, the goal here
> isn't actually to make pg_basebackup into an enterprise backup tool,
> it's to make things easier for the external tools to do block-level
> backups.

Well, I agree with you that the goal is not to make pg_basebackup an
enterprise backup tool.  However, I don't see teaching it to take
incremental backups as opposed to that goal.  I think backup
management and retention should remain firmly outside the purview of
pg_basebackup and left either to some other in-core tool or maybe even
to out-of-core tools.  However, I don't see any reason why that the
task of taking an incremental and/or parallel backup should also be
left to another tool.

There is a very close relationship between the thing that
pg_basebackup already does (copy everything) and the thing that we
want to do here (copy everything except blocks that we know haven't
changed). If we made it the job of some other tool to take parallel
and/or incremental backups, that other tool would need to reimplement
a lot of things that pg_basebackup has already got, like tar vs. plain
format, fast vs. spread checkpoint, rate-limiting, compression levels,
etc.  That seems like a waste.  Better to give pg_basebackup the
capability to do those things, and then any backup management tool
that anyone writes can take advantage of those capabilities.

I come at this, BTW, from the perspective of having just spent a bunch
of time working on EDB's Backup And Recovery Tool (BART).  That tool
works in exactly the manner you seem to be advocating: it knows how to
do incremental and parallel full backups, and it also does backup
management.  However, this has not turned out to be the best division
of labor.  People who don't want to use the backup management
capabilities may still want the parallel or incremental backup
capabilities, and if all of that is within the envelope of an
"enterprise backup tool," they don't have that option.  So I want to
split it up.  I want pg_basebackup to take all the kinds of backups
that PostgreSQL supports -- full, incremental, parallel, serial,
whatever -- and I want some other tool -- pgBackRest, BART, barman, or
some yet-to-be-invented core thing to do the management of those
backups.  Then everybody can use exactly the bits they want.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company