Re: Logical replication and multimaster

Excellent.

I just pushed a change to pglogical_output that exposes the row contents (and the rest of the reorder change buffer contents) to hooks that want it, by the way.

 

Yep, that'll tide you over. We're working hard on getting the downstream part ready and you'll find it more flexible.

 

That's not too surprising, given that you'll have significant overheads for checking if keys are present when doing updates.

Yep, that's the idea.

 

I'm curious about what you're using the 'isLocal' field for.

For MM you should only need to examine the replication origin assigned to the transaction to determine whether you're going to forward it or not.

Were you not able to achieve what you wanted with a hook? If not, then we might need another hook. Could you explain what it's for in more detail?

What I suggest is: have your downstream client install a pglogical_output hook for the transaction filter hook. There, examine the replication origin passed to the hook. If you want to forward locally originated xacts only (such as for mesh multimaster) you can just filter out everything where the origin is not InvalidRepOriginId. There are example hooks in contrib/pglogical_output_plhooks .

There'll be a simple MM example using filter hooks in the pglogical downstream btw and we're working hard to get that out.

 

Yes, I agree that it should be passed to the API for the output protocol. It's pretty harmless. Please feel free to send a pull req.

Note that we haven't made that pluggable from the outside though; there's no way to load a new protocol distributed separately from pglogical_output. The idea is really to make sure that between the binary protocol and json protocol we meet the reasonably expected set of use cases and don't need pluggable protocols. Perhaps that's over-optimistic, but we've already got and output plugin that has plug-in hooks, a plugin for a plugin. Do we need another? Also, if we allow dynamic loading of new protocols then that means we'll have a much harder time changing the protocol implementation API later, so it's not something I'm keen to do. Also, to make it secure to allow users to specify the protocol we'd have to make protocols implement an extension with a C function in pg_proc to return its API struct, like we do for hooks. So there'd be more hoop-jumping required to figure out how to talk to the client.

If possible I'd like to find any oversights and omissions in the current protocol and its APIs to meet future use cases without having to introduce protocol plugins for an output plugin.

Yeah, we weren't really thinking of the protocol API as intended to be pluggable and extensible. If you define new protocols you have to change the rest of the output plugin code anyway.

Lets look at what protocol changes are needed to address your use case and see whether it's necessary to take the step of making the protocol fully pluggable or not.