RE: Proposed WAL changes

> > It seems that you want guarantee more than me, Tom -:)
> 
> No, but I want a system that's not brittle. You seem to be content to
> design a system that is reliable as long as the WAL log is OK 
> but loses the entire database unrecoverably as soon as one bit goes bad
> in the log.

I don't see how absence of old checkpoint forces losing entire database.
You probably will get better consistency by re-applying modifications
which supposed to be in data files already but it seems questionable
to me.

> > BTW, in some my tests size of on-line logs was ~ 200Mb with default
> > checkpoint interval. So, it's worth to care about on-line logs size.
> 
> Okay, but to me that suggests we need a smarter log management strategy,
> not a management strategy that throws away data we might wish we still
> had (for manual analysis if nothing else).

This is what should be covered by archiving log files. Unimplemented -:(

> Perhaps the checkpoint creation rule should be "every M seconds *or*
> every N megabytes of log, whichever comes first".

I like this! Regardless usability of keeping older checkpoint (especially
in future, with log archiving) your rule is worth in any case.
(Nevertheless, keeping two checkpoints still increases disk requirements -:)
But seems I have to waive my objection if I didn't convince you - it's
really simplest way to get WAL restart-ability and I personally have no
ability to implement log scanning now).

> > Please convince me that NEXTXID is necessary.
> > Why add anything that is not useful?
> 
> I'm not convinced that it's not necessary. In particular, 
> consider the case where we are trying to recover from a crash using
> an on-line checkpoint as our last readable WAL entry. In the pre-NEXTXID
> code, this checkpoint would contain the current XID counter and an
> advanced-beyond-current OID counter. I think both of those numbers should
> be advanced beyond current, so that there's some safety margin against
> reusing XIDs/OIDs that were allocated by now-lost XLOG entries.
> The OID code is doing this right, but the XID code wasn't.
> 
> Again, it's a question of brittleness.  Yes, as long as everything
> operates as designed and the WAL log never drops a bit, we don't need
> it.  But I want a safety margin for when things aren't perfect.

Once again - my point is that in the event of lost log record one shouldn't
try to use existent database but just dump it, etc - ie no reason to keep
info about allocated XIDs. But keeping NEXTXID costs nothing, at least -:)

Vadim