Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 14:08, Alvaro Hernandez <aht@ongres.com> wrote:

 

    OK, let me try to do that. I believe data integration is a priority.

Definitely agree so far.

 

- If you want to develop your own output plugin, then your market is reduced as you have to exclude all the managed solutions (until, and only if, you would convince them to include your plugin... highly unlikely, very difficult). And probably another % of enterprise environments which will hesitate to link your own plugin to the running production database. Last but not least, you need to compile and test (and testing this is far from easy) on multiple OSes/architectures.

Right. You probably don't want one output plugin per application.

This doesn't mean it has to be in-core, just flexible and share-able by many, and adopted by cloud vendors. Like say PostGIS.

    That would be nice. But this is chicken-and-egg: an out-of-core plugin won't probably be used by many if applications like the ones I was mentioning if they do not exist. And developing such an application is so much less interesting if a significant part of your market is excluded from your app.

    However, it could work the other way around: a sufficiently good enough in-core base plugin could foster applications/ecosystem, which once adopted by users could push much more easily for other more advanced out-of-core plugins, that would be more easily accepted by pressure as those tools would already be with significant traction. But I don't see it the other way around.

 

- If you stick to in-core plugins, then you need to support at least three different output formats if you want to support 9.4+: test_decoding (and pray it works!), pgoutput, and the "new" in-core plugin that was proposed at the beginning of this thread, if that would see the light.

The only practical way will IMO be to have whatever new plugin it also have an out-of-core version maintained for older Pg versions, where it can be installed.

  

But only in-core plugins help for general-purpose solutions.

I still don't agree there. If there's enough need/interest/adoption you can get cloud vendors on board, they'll feel the customer pressure. It's not our job to create that pressure and do their work for them.

    Don't want to get into a loop, but as I said before it's chicken-and-egg. But nobody is asking core to do their work. As much as I love it, I think logical decoding is a bit half-baked until there is a single, quality, in-core plugin, as it discourages its usage, because of the reasons I stated.

I see nothing wrong with a plugin starting off out of core and being adopted+adapted later, assuming it's done well.

That said, I'm all in favour of a generic json output plugin that shares infrastructure with logical replication, so people who are on inflexible environments have a fallback option. I just don't care to write it.

    That's better than nothing. But as much as interoperable json may be, people still need to talk the (binary) replication protocol to use it. So once you talk binary protocol, why not talk binary also for the output plugin and have a much more efficient output? Again, nothing against json, but if a new plugin would be included in-core, I'd say json + binary also. Or just document pgoutput, as it could be good enough.

    Cheers,

    Álvaro

-- 

Alvaro Hernandez


-----------
OnGres