Re: Changeset Extraction Interfaces

On 2014-01-07 17:54:21 +0100, Andres Freund wrote:
> On 2013-12-12 16:49:33 +0100, Andres Freund wrote:
> > On 2013-12-12 10:01:21 -0500, Robert Haas wrote:
> > > On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> > > > As far as I have been thinking of, this would be another catalog table like
> > > > pg_decoding_plugin(oid, dpname name, dpload regproc).
> > > 
> > > Instead of adding another catalog table, I think we should just define
> > > a new type.  Again, please look at the way that foreign data wrappers
> > > do this:
> > 
> > I don't really see what the usage of a special type has to do with this,
> > but I think that's besides your main point. What you're saying is that
> > the output plugin is just defined by a function name, possibly schema
> > prefixed. That has an elegance to it. +1
> 
> Ok, so I've implemented this, but I am not so sure it's sufficient,
> there's some issue:
> Currently a logical replication slot has a plugin assigned, previously
> that has just been identified by the basename of a .so. But with the
> above proposal the identifier is pointing to a function, currently via
> its oid. But what happens if somebody drops or recreates the function?
> We can't make pg_depend entries or anything since that won't work on a
> standby.
> Earlier, if somebody removed the .so we'd just error out, but pg's
> dependency tracking always only mattered to things inside the catalogs.
> 
> I see the following possible solutions for this:
> 
> 1) accept that fact, and throw an error if the function doesn't exist
> anymore, or has an unsuitable signature. We can check the return type of
> output_plugin_callbacks, so that's a pretty specific test.
> 
> 2) Create a pg_output_plugin catalog and prevent DROP OUTPUT PLUGIN (or
> similar) when there's a slot defined. But how'd that work if the slot is
> only defined on standbys? We could have the redo routine block and/or
> kill the slot if necessary?
> 
> 3) Don't assign a specific output plugin to a slot, but have it
> specified everytime data is streamed, not just when a slot is
> created. Currently that wouldn't be a problem, but I am afraid it will
> constrict some future optimizations.
> 
> Good ideas?

So, Robert and I had a IM discussion about this. Neither of us was
particularly happy about the proposed solutions.

So, what we've concluded is that using a function as the handler doesn't
work out well enough given the constraints (primarily the inability to
create dependency records on a HS node). We've concluded that the best
way forward is a variant of the current implementation where the output
plugin is specified as a dynamic library. Which is:
CREATE_REPLICATION_SLOT slot_name LOGICAL OUTPUT_PLUGIN library_name;
but in contrast to the current code where each individual output plugin
callback is dlsym()ed via a fixed function name, only a
_PG_output_plugin_init() function is looked up & called which fills out
a struct containing the individual callbacks. Additionally the "init"
and "cleanup" output plugin callbacks will be renamed to
startup/shutdown to avoid possible confusions.

This unfortunately still prohibits implementing output plugins within
the core postgres binary, but that can be solved by shipping
core-provided output plugins - should they ever exist - as shared
objects, like it's already done for libpqwalreceiver.

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services