Re: [RFC] Incremental backup v2: add backup profile to base backup

On 10/06/2014 06:33 PM, Marco Nenciarini wrote:
> Il 03/10/14 22:47, Robert Haas ha scritto:
>> 2. Take a differential backup.  In the backup label file, note the LSN
>> of the fullback to which the differential backup is relative, and the
>> newest LSN guaranteed to be present in the differential backup.  The
>> actual backup can consist of a series of 20-byte buffer tags, those
>> being the exact set of blocks newer than the base-backup's
>> latest-guaranteed-to-be-present LSN.  Each buffer tag is followed by
>> an 8kB block of data.  If a relfilenode is truncated or removed, you
>> need some way to indicate that in the backup; e.g. include a buffertag
>> with forknum = -(forknum + 1) and blocknum = the new number of blocks,
>> or InvalidBlockNumber if removed entirely.
>
> To have a working backup you need to ship each block which is newer than
> latest-guaranteed-to-be-present in full backup and not newer than
> latest-guaranteed-to-be-present in the current backup. Also, as a
> further optimization, you can think about not sending the empty space in
> the middle of each page.
>
> My main concern here is about how postgres can remember that a
> relfilenode has been deleted, in order to send the appropriate "deletion
> tag".
>
> IMHO the easiest way is to send the full list of files along the backup
> and let to the client the task to delete unneeded files. The backup
> profile has this purpose.

Right, but the server doesn't need to send a separate backup profile 
file for that. Rather, anything that the server *didn't* send, should be 
deleted.

I think the missing piece in this puzzle is that even for unmodified 
blocks, the server should send a note saying the blocks were present, 
but not modified. So for each file present in the server, the server 
sends a block stream. For each block, it sends either the full block 
contents, if it was modified, or a simple indicator that it was not 
modified.

There's a downside to this, though. The client has to read the whole 
stream, before it knows which files were present. So when applying a 
block stream directly over an old backup, the client cannot delete files 
until it has applied all the other changes. That needs more needs more 
disk space. With a separate profile file that's sent *before* the rest 
of the backup, you could delete the obsolete files first. But that's not 
a very big deal. I would suggest that you leave out the profile file in 
the first version, and add it as an optimization later, if needed.

> Moreover, I do not like the idea of using only a stream of block as the
> actual differential backup, for the following reasons:
>
> * AFAIK, with the current infrastructure, you cannot do a backup with a
> block stream only. To have a valid backup you need many files for which
> the concept of LSN doesn't apply.

Those should be sent in whole. At least in the first version. The 
non-relation files are small compared to relation files, so it's not too 
bad to just include them in full.

>> 3. Apply a differential backup to a full backup to create an updated
>> full backup.  This is just a matter of scanning the full backup and
>> the differential backup and applying the changes in the differential
>> backup to the full backup.
>>
>> You might want combinations of these, like something that does 2+3 as
>> a single operation, for efficiency, or a way to copy a full backup and
>> apply a differential backup to it as you go.  But that's it, right?
>> What else do you need?
>
> Nothing else. Once we agree on definition of involved files and
> protocols formats, only the actual coding remains.

BTW, regarding the protocol, I have an idea. Rather than invent a whole 
new file format to represent the modified blocks, can we reuse some 
existing binary diff file format? For example, the VCDIFF format (RFC 
3284). For each unmodified block, the server would send a vcdiff COPY 
instruction, to "copy" the block from the old backup, and for a modified 
block, the server would send an ADD instruction, with the new block 
contents. The VCDIFF file format is quite flexible, but we would only 
use a small subset of it. I believe that subset would be just as easy to 
generate in the backend as a custom file format, but you could then use 
an external tool (xdelta3, open-vcdiff) to apply the diff manually, in 
case of emergency. In essence, the server would send a tar stream as 
usual, but for each relation file, it would send a VCDIFF file with name 
"<relfilenode>.vcdiff" instead.

- Heikki