Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 19:56, Andrew Dunstan wrote:
>
> On 09/25/2017 12:48 PM, Alvaro Hernandez wrote:
>>
>> On 25/09/17 19:39, Petr Jelinek wrote:
>>> Well, test_decoding is not meant for production use anyway, no need for
>>> middleware to support it. The pgoutput is primarily used for internal
>>> replication purposes, which is why we need something with more
>>> interoperability in mind in the first place. The new plugin should still
>>> support publications etc though IMHO.
>>>
>>>>       However, having said that, and while json is a great output format
>>>> for interoperability, if there's a discussion on which plugin to
>>>> include
>>>> next, I'd also favor one that has some more compact representation
>>>> format (or that supports several formats, not only json).
>>>>
>>> JSON is indeed great for interoperability, if you want more compact
>>> format, use either pgoutput or write something of your own or do
>>> conversion to something else in your consumer. I don't think postgres
>>> needs to provide 100 different formats out of the box when there is an
>>> API. The JSON output does not have to be extremely chatty either btw.
>>>
>>      In my opinion, logical decoding plugins that don't come with core
>> are close to worthless (don't get me wrong):
>>
>> - They very unlikely will be installed in managed environments (an
>> area growing significantly).
>> - As anything that is not in core, raises concerns by users.
>> - Distribution and testing are non-trivial: many OS/archs combinations.
>>
>>      Given the above, I believe having a general-purpose output plugin
>> in-core is critical to the use of logical decoding. As for 9.4-9.6
>> there is test_decoding, and given that AWS uses it for production,
>> that's kind of fine. For 10 there is at least pgoutput, which could be
>> used (even though it was meant for replication). But if a new plugin
>> is to be developed for 11+, one really general purpose one, I'd say
>> json is not a good choice if it is the only output it would support.
>> json is too verbose, and replication, if anything, needs performance
>> (it is both network heavy and serialization/deserialization is quite
>> expensive). Why not, if one and only one plugin would be developed for
>> 11+, general purpose, do something that is, indeed, more general,
>> i.e., that supports high-performance scenarios too?
>>
>>
>>    
>
> A general purpose lower bandwidth plugin might one supporting Protocol
> Buffers. The downside is that unlike json it's not self-contained, you
> need the message definitions to interpret the stream, AIUI.
    Sure. But that's just a matter of documenting them, or even better, 
providing the .proto files, which are language-independent.
    There are also many other efficient serialization formats to 
explore, some self-contained, some not.

    Álvaro

-- 

Alvaro Hernandez


-----------
OnGres



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers