Re: Are file system level differential/incremental backups possible?

On 10/20/2011 02:00 AM, Bob Hatfield wrote:
>>> Is it possible to do a full file system level backup of the data
>>> directory, say once a week, and differentials or incrementals daily?
>>
>> I'd love to be able to do this, but you can't do it usefully at a
>> file-system level. There's too much churn in the data files for even a
>> binary diff to be much use - and even if it were, the performance of it
>> would be miserable.
>
> Actually, for us, a full backup is currently about 950GB which takes
> about 24 hours and a diff several days later is only 150GB and takes
> two hours.  (It takes significantly less time not only becuase it's
> less data but also because the diff job doesn't have to compete with
> other backup jobs (since it's faster due to less data).)

Wow. I'm very surprised by that, but it's good to hear. I would've
expected a much poorer result, and have never seen anything like that
good a result when I've tested with my (much smaller) data sets.

Perhaps a binary diff based backup mechanism that's aware of Pg's
structure and can skip holes, etc is worth looking into after all.

--
Craig Ringer