Re: WAL format changes

On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote:
> On 18.06.2012 21:08, Heikki Linnakangas wrote:
> > On 18.06.2012 21:00, Robert Haas wrote:
> >> On Thu, Jun 14, 2012 at 5:58 PM, Andres Freund<andres@2ndquadrant.com>
> >> 
> >> wrote:
> >>>> 1. Use a 64-bit segment number, instead of the log/seg combination.
> >>>> And don't waste the last segment on each logical 4 GB log file. The
> >>>> concept of a "logical log file" is now completely gone. XLogRecPtr is
> >>>> unchanged,
> >>>> but it should now be understood as a plain 64-bit value, just split
> >>>> into
> >>>> two 32-bit integers for historical reasons. On disk, this means that
> >>>> there will be log files ending in FF, those were skipped before.
> >>> 
> >>> Whats the reason for keeping that awkward split now? There aren't
> >>> that many
> >>> users of xlogid/xcrecoff and many of those would be better served by
> >>> using
> >>> helper macros.
> >> 
> >> I wondered that, too. There may be a good reason for keeping it split
> >> up that way, but we at least oughta think about it a bit.
> > 
> > The page header contains an XLogRecPtr (LSN), so if we change it we'll
> > have to deal with pg_upgrade. I guess we could still keep XLogRecPtr
> > around as the on-disk representation, and convert between the 64-bit
> > integer and XLogRecPtr in PageGetLSN/PageSetLSN. I can try that out -
> > many xlog calculations would admittedly be simpler if it was an uint64.
> 
> Well, that was easier than I thought. Attached is a patch to make
> XLogRecPtr a uint64, on top of my other WAL format patches. I think we
> should go ahead with this.
Cool. You plan to merge XLogSegNo with XLogRecPtr in that case? I am not sure 
if having two representations which just have a constant factor inbetween 
really makes sense.

> The LSNs on pages are still stored in the old format, to avoid changing
> the on-disk format and breaking pg_upgrade. The XLogRecPtrs stored the
> control file and WAL are changed, however, so an initdb (or at least
> pg_resetxlog) is required.
Sounds sensible.

> Should we keep the old representation in the replication protocol
> messages? That would make it simpler to write a client that works with
> different server versions (like pg_receivexlog). Or, while we're at it,
> perhaps we should mandate network-byte order for all the integer and
> XLogRecPtr fields in the replication protocol.
The replication protocol uses pq_sendint for integers which should take care 
of converting to big endian already. I don't think anything but the wal itself 
is otherwise transported in a binary fashion? So I don't think there is any 
such architecture dependency in the protocol currently?

I don't really see a point in keeping around a backward-compatible 
representation just for the sake of running such tools on multiple versions. I 
might not be pragmatic enough, but: Why would you want to do that *at the 
moment*? Many of the other tools are already version specific, so...
When the protocol starts to be used by more tools, maybe, but imo were not 
there yet.

But then its not hard to convert to the old representation for that.

> I kept the %X/%X representation in error messages etc. I'm quite used to
> that output, so reluctant to change it, although it's a bit silly now
> that it represents just 64-bit value. Using UINT64_FORMAT would also
> make the messages harder to translate.
No opinion on that. Its easier to see for me whether two values are exactly 
the same or very similar with the 64bit representation but its harder to gauge 
bigger differences. So ...

Greetings,

Andres
-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services