Re: parallelizing the archiver

On 10/22/21, 4:35 PM, "Robert Haas"  wrote:
> I was imagining something like what logical decoding does. In that
> case, you make a _PG_output_plugin_init function and it returns a
> table of callbacks. Then the core code invokes those callbacks at the
> appropriate times.

Here is an attempt at doing this.  Archive modules are expected to
declare _PG_archive_module_init(), which can define GUCs, register
background workers, etc.  This function must at least define the
archive callbacks.  For now, I've introduced two callbacks.  The first
is for checking that the archive module is configured, and the second
contains the actual archiving logic.

I've written this so that the same library can be used for multiple
purposes (e.g., it could be in shared_preload_libraries and
archive_library).  I don't know if that's really necessary, but it
seemed to me like a reasonable way to handle the changes to the
library loading logic that we need anyway.

0002 is still a sample backup module, but I also added some handling
for preexisting archives.  If the preexisting archive file has the
same contents as the current file to archive, archiving is allowed to
continue.  If the contents don't match, archiving fails.  This sample
module could still produce unexpected results if two servers were
sending archives to the same directory.  I stopped short of adding
handling for that case, but that might be a good thing to tackle next.

Nathan