Re: finding changed blocks using WAL scanning

On Sat, Apr 20, 2019 at 04:17:08PM -0400, Robert Haas wrote:
> On Sat, Apr 20, 2019 at 9:18 AM Bruce Momjian  wrote:
> > > I think you've got to prevent the WAL from being removed until a
> > > .modblock file has been written.  In more detail, you should (a) scan
> > > all the WAL segments that will be summarized in the .modblock file,
> > > (b) write the file under a temporary name, (c) fsync it, (d) rename it
> > > into place, (e) fsync it again, and (f) then allow those WAL segments
> > > to be removed, if they are otherwise eligible to be removed.
> >
> > Makes sense.  So when you are about to remove WAL, you create the
> > .modblock files for all complete WAL files and only create a new one
> > when you are about to remove a WAL that was not in a previous .modblock
> > file.
> 
> There will often be a partial WAL record at the end of each file.  So
> if you make a .modblock file for WAL files 1-10, you can probably
> remove files 1-9, but you probably have to keep WAL file 10 around
> until you generate the NEXT .modblock file, because otherwise you
> wouldn't be able to read and parse the WAL record that spans the end
> of file 10 and the beginning of file 11.
> 
> This is a detail that is very very very important to get right.

Good point.  You mentioned:

	It seems better to me to give the files names like
	${TLI}.${STARTLSN}.${ENDLSN}.modblock, e.g.
	00000001.0000000168000058.00000001687DBBB8.modblock, so that you can
	see exactly which *records* are covered by that segment.

but it seems like it should be ${TLI}.${ENDLSN}... (END first) because
you would not want to delete the modblock file until you are about to
delete the final WAL, not the first WAL, but as you mentioned, it might
be ENDLSN-1.

-- 
  Bruce Momjian          http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +