Обсуждение: Validity of using the test_decoding plugin for production?

Hi all,

I'm considering using the test_decoding[1] plugin in conjunction with
Amazon's recent update for RDS with PostgreSQL[2] which includes logical
replication support. The concern I currently have is the stability and
output format of the test_decoding plugin. Specifically if it's built to
handle general purpose replication, has the stability to perform in a
production environment, and if there are any concerns about depending on
it for critical infrastructure given my current understanding is that
new output plugins cannot be used with RDS.

Comments to the above or experiences using test_decoding in a production
setup would be appreciated, whether inside RDS or not.

Thanks,

-Josh

[1]: https://www.postgresql.org/docs/9.6/static/test-decoding.html
[2]:
https://aws.amazon.com/blogs/aws/amazon-rds-for-postgresql-new-minor-versions-logical-replication-dms-and-more/

On Tue, Nov 1, 2016 at 12:57 AM, Joshua Kehn <josh@kehn.us> wrote:
> I'm considering using the test_decoding[1] plugin in conjunction with
> Amazon's recent update for RDS with PostgreSQL[2] which includes logical
> replication support. The concern I currently have is the stability and
> output format of the test_decoding plugin. Specifically if it's built to
> handle general purpose replication, has the stability to perform in a
> production environment, and if there are any concerns about depending on
> it for critical infrastructure given my current understanding is that
> new output plugins cannot be used with RDS.
>
> Comments to the above or experiences using test_decoding in a production
> setup would be appreciated, whether inside RDS or not.

Personally, I have an image of test_decoding as something people can
use as a base to implement their own decoder plugin to satisfy their
requirements. Based on that, it is not something that would be used
for general purposes regarding replication to replicate logical
changes. How are you going to use it anyway? First deparse its output
and adapt it to your needs? You would give yourself a favor to hack
your own thing, and that would finish by being more performant as
their is not need to do a post-receive format change of the data
received by a single change. Now regarding the stability of the thing,
the code is in a pretty nice shape.
--
Michael

> On Tue, Nov 1, 2016, at 04:00 AM, Michael Paquier wrote:
> How are you going to use it anyway? First deparse its output
> and adapt it to your needs?

That's the intention. Take the output of test_decoding, transform it
into a reasonable object form, and feed that into an events stream for
real-time consumption.

The initial question was suitability of the test_decoding plugin, given
that it is the only one I see available on RDS and requiring a switch to
a self-managed Postgres instance—to use a separate plugin—would kill
this idea.

Are there any success/failure stories about using logical decoding as a
way of generating an events stream? There's some new question marks over
the consumer side, whether to use pg_recvlogical or something with a
peek/write to stream/pop cycle to protect the consumer from failure.

Given a single reader, what is the common failure mode when consuming
the output?

Best,

-Josh

On Wed, Nov 2, 2016 at 4:01 AM, Joshua Kehn <josh@kehn.us> wrote:
> That's the intention. Take the output of test_decoding, transform it
> into a reasonable object form, and feed that into an events stream for
> real-time consumption.

Hm...

> The initial question was suitability of the test_decoding plugin, given
> that it is the only one I see available on RDS and requiring a switch to
> a self-managed Postgres instance—to use a separate plugin—would kill
> this idea.

OK, and no custom binaries can be installed in this case. IMO you
would have a far easier time moving to something else if you are
willing to fetch custom changes or to use some of your own plugins.
Vendor-locking with RDS is your problem here, and trying to use the
stream from test_decoding to then modify it in the way you want is
likely going to embark you in a journey you'll regret in the long
term. I don't know the opinion of others on the matter... But IMO
test_decoding is an example of output plugin, so there is no need to
keep its output stable across major releases if any needs for another
patch or feature pops in.

> Are there any success/failure stories about using logical decoding as a
> way of generating an events stream? There's some new question marks over
> the consumer side, whether to use pg_recvlogical or something with a
> peek/write to stream/pop cycle to protect the consumer from failure.

pglogical has been one solution around that worked on logical
decoding, and as far as I recall it uses its own protocol to replicate
changes. I never had a look at it in details but that may be something
to look into if you are looking for some story in this area.
--
Michael