Re: [DESIGN] Incremental checksums

On 7/13/15 4:02 PM, David Christensen wrote:
>
>> On Jul 13, 2015, at 3:50 PM, Jim Nasby <Jim.Nasby@BlueTreble.com> wrote:
>>
>> On 7/13/15 3:26 PM, David Christensen wrote:
>>> * Incremental Checksums
>>>
>>> PostgreSQL users should have a way up upgrading their cluster to use data checksums without having to do a costly
pg_dump/pg_restore;in particular, checksums should be able to be enabled/disabled at will, with the database enforcing
thelogic of whether the pages considered for a given database are valid.
 
>>>
>>> Considered approaches for this are having additional flags to pg_upgrade to set up the new cluster to use checksums
wherethey did not before (or optionally turning these off).  This approach is a nice tool to have, but in order to be
ableto support this process in a manner which has the database online while the database is going throught the initial
checksumprocess.
 
>>
>> It would be really nice if this could be extended to handle different page formats as well, something that keeps
rearingit's head. Perhaps that could be done with the cycle idea you've described.
 
>
> I had had this thought too, but the main issues I saw were that new page formats were not guaranteed to take up the
samespace/storage, so there was an inherent limitation on the ability to restructure things out *arbitrarily*; that
beingsaid, there may be a use-case for the types of modifications that this approach *would* be able to handle.
 

After some discussion on IRC, I there's 2 main points to consider.

First, we're currently unhappy with how relfrozenxid works, and this 
proposal follows the same pattern of having essentially a counter field 
in pg_class. Perhaps this is OK because things like checksum really 
shouldn't change that often. (My inclination is that fields in pg_class 
are OK for now.)

Second, there are 4 use cases here that are very similar. We should 
*consider* them now, while designing this. That doesn't mean the first 
patch needs to support anything other than checksums.

1) Page layout changes
2) Page run-time changes (currently only checksums)
3) Tuple layout changes (ie: HEAP_MOVED_IN)
4) Tuple run-time changes (ie: DROP COLUMN)

1 is currently handled in pg_upgrade by forcing a page-by-by-page copy 
during upgrade. Doing this online would require the same kind of 
conversion plugin pg_upgrade uses. If we want to support conversions 
that need extra free space on a page we'd also need support for that.

2 is similar to 1, except this can change via GUC or similar. Checksums 
are an example of this, as is creating extra free space on a page to 
support an upgrade.

3 & 4 are tuple-level equivalents to 1 & 2.

I think the bigger challenge to these things is how to track the status 
of a conversion (as opposed to the conversion function itself).

- Do we want each of these to have a separate counter in pg_class? 
(rellastchecksum, reloldestpageversion, etc)

- Should that info be combined?
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Data in Trouble? Get it in Treble! http://BlueTreble.com