Re: block-level incremental backup

On Wed, Apr 24, 2019 at 12:57 PM Stephen Frost <sfrost@snowman.net> wrote:
> So, I had a thought about that when I was composing the last email and
> while I'm still unsure about it, maybe it'd be useful to mention it
> here- do we really need a list of every *file*, or could we reduce that
> down to a list of relations + forks for the main data directory, and
> then always include whatever other directories/files are appropriate?

I'm not quite sure what the difference is here.  I agree that we could
try to compact the list of file names by saying 16384 (24 segments)
instead of 16384, 16384.1, ..., 16384.23, but I doubt that saves
anything meaningful.  I don't see how we can leave anything out
altogether.  If there's a filename called boaty.mcboatface in the
server directory, I think we've got to back it up, and that won't
happen unless the client knows that it is there, and it won't know
unless we include it in a list.

> When it comes to operating in chunks, well, if we're getting a list of
> relations instead of files, we do have this thing called cursors..

Sure... but they don't work for replication commands and I am
definitely not volunteering to change that.

> I would think the client would be able to just ask for the list of
> modified files, when it comes to building up the list of files to ask
> for, which could potentially be done based on mtime instead of by WAL
> scanning or by scanning the files themselves.  Don't get me wrong, I'd
> prefer that we work based on the WAL, since I have more confidence in
> that, but certainly quite a few of the tools do work off mtime these
> days and while it's not perfect, the risk/reward there is pretty
> palatable to a lot of people.

That approach, as with a few others that have been suggested, requires
that the client have access to the previous backup, which makes me
uninterested in implementing it.  I want a version of incremental
backup where the client needs to know the LSN of the previous backup
and nothing else.  That way, if you store your actual backups on a
tape drive in an airless vault at the bottom of the Pacific Ocean, you
can still take incremental backup against them, as long as you
remember to note the LSNs before you ship the backups to the vault.
Woohoo!  It also allows for the wire protocol to be very simple and
the client to be very simple; neither of those things is essential,
but both are nice.

Also, I think using mtimes is just asking to get burned.  Yeah, almost
nobody will, but an LSN-based approach is more granular (block level)
and more reliable (can't be fooled by resetting a clock backward, or
by a filesystem being careless with file metadata), so I think it
makes sense to focus on getting that to work.  It's worth keeping in
mind that there may be somewhat different expectations for an external
tool vs. a core feature.  Stupid as it may sound, I think people using
an external tool are more likely to do things read the directions, and
those directions can say things like "use a reasonable filesystem and
don't set your clock backward."  When stuff goes into core, people
assume that they should be able to run it on any filesystem on any
hardware where they can get it to work and it should just work.  And
you also get a lot more users, so even if the percentage of people not
reading the directions were to stay constant, the actual number of
such people will go up a lot. So picking what we seem to both agree to
be the most robust way of detecting changes seems like the way to go
from here.

> I suspect some of that's driven by how they get solved and if we decide
> we have to solve all of them.  With things like MAX_RATE + incremental
> backups, I wonder how that's going to end up working, when you have the
> option to apply the limit to the network, or to the disk I/O.  You might
> have addressed that elsewhere, I've not looked, and I'm not too
> particular about it personally either, but a definition could be "max
> rate at which we'll read the file you asked for on this connection" and
> that would be pretty straight-forward, I'd think.

I mean, it's just so people can tell pg_basebackup what rate they want
via a command-line option and have it happen like that.  They don't
care about the rates for individual files.

> > Issue #1: If you manually add files to your backup, remove files from
> > your backup, or change files in your backup, bad things will happen.
> > There is fundamentally nothing we can do to prevent this completely,
> > but it may be possible to make the system more resilient against
> > ham-handed modifications, at least to the extent of detecting them.
> > That's maybe a topic for another thread, but it's an interesting one:
> > Andres and I were brainstorming about it at some point.
>
> I'd certainly be interested in hearing about ways we can improve on
> that.  I'm alright with it being on another thread as it's a broader
> concern than just what we're talking about here.

Might be a good topic to chat about at PGCon.

> > Issue #2: You can only restore an LSN-based incremental backup
> > correctly if you have a base backup whose start-of-backup LSN is
> > greater than or equal to the threshold LSN used to take the
> > incremental backup.  If #1 is not in play, this is just a simple
> > cross-check at restoration time: retrieve the 'START WAL LOCATION'
> > from the prior backup's backup_label file and the threshold LSN for
> > the incremental backup from wherever you decide to store it and
> > compare them; if they do not have the right relationship, ERROR.  As
> > to whether #1 might end up in play here, anything's possible, but
> > wouldn't manually editing LSNs in backup metadata files be pretty
> > obviously a bad idea?  (Then again, I didn't really think the whole
> > backup_label thing was that confusing either, and obviously I was
> > wrong about that.  Still, editing a file requires a little more work
> > than removing it... you have to not only lie to the system, you have
> > to decide which lie to tell!)
>
> Yes, that'd certainly be at least one cross-check, but what if you've
> got an incremental backup based on a prior incremental backup that's
> based on a prior full, and you skip the incremental backup inbetween
> somehow?  Or are we just going to state outright that we don't support
> incremental-on-incremental (in which case, all backups would actually be
> either 'full' or 'differential' in the pgBackRest parlance, anyway, and
> that parlance comes from my recollection of how other tools describe the
> different backup types, but that was from many moons ago and might be
> entirely wrong)?

I have every intention of supporting that case, just as I described in
my original email, and the algorithm that I just described handles it.
You just have to repeat the checks for every backup in the chain.   If
you have a backup A, and a backup B intended as an incremental vs. A,
and a backup C intended as an incremental vs. B, then the threshold
LSN for C is presumably the starting LSN for B, and the threshold LSN
for B is presumably the starting LSN for A.  If you try to restore
A-B-C you'll check C vs. B and find that all is well and similarly for
B vs. A.  If you try to restore A-C, you'll find out that A's start
LSN precedes C's threshold LSN and error out.

> Even if what we're talking about here is really only "differentials", or
> backups where the incremental contains all the changes from a prior full
> backup, if the only check is "full LSN is greater than or equal to the
> incremental backup LSN", then you have a potential problem that's larger
> than just the incrementals no longer being valid because you removed the
> full backup on which they were taken- you might think that an *earlier*
> full backup is the one for a given incremental and perform a restore
> with the wrong full/incremental matchup and end up with a corrupted
> database.

No, the proposed check is explicitly designed to prevent that.  You'd
get a restore failure (which is not great either, of course).

> management.  The idea of implementing block-level incrementals while
> pushing the backup management, expiration, and dependency between
> incrementals and fulls on to the user to figure out just strikes me as
> entirely backwards and, frankly, to be gratuitously 'itch scratching' at
> the expense of what users really want and need here.

Well, not everybody needs or wants the same thing.  I wouldn't be
proposing it if my employer didn't think it was gonna solve a real
problem...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company