Обсуждение: Logical decoding
Hi,
I have a question about logical decoding of Postgres.
where are the entry points to logical decoding?
Specifically, we want to know whether logical decoding happens immediately after commit, or whether there is a polling thread that scans the Write Ahead Log and then dumps to the special table?
and where are this code is in the codebase?
Thanks,
Joshua
On 9 July 2016 at 01:57, Joshua Bay <joshuabay93@gmail.com> wrote:
src/backend/replication/logical/*
-- where are the entry points to logical decoding?
Well, it depends on what level you're looking at.
Data is read using the xlogreader and fed into the decoding system by the walsender (when using a logical slot) and the SQL level pg_logical_slot_[get|peek]_changes functions. See XLogSendLogical() as called by WalSndLoop() and see pg_logical_slot_get_changes_guts() .
WAL records are passed through to logical decoding and depending on the xlog entry type the reorder buffer and/or snapshot builder. Then when a transaction commit is detected the registered output plugin is invoked and its callbacks are used to process the buffered transaction.
Specifically, we want to know whether logical decoding happens immediately after commit, or whether there is a polling thread that scans the Write Ahead Log and then dumps to the special table?
There's no "polling thread", but that's probably the closer of the two. A walsender using a logical slot reads WAL as it becomes available. It sleeps on a latch if it runs out of WAL to read and is woken when there's more. It isn't dumped to some special table, it's managed by the reorder buffer infrastructure which uses its own special data structures. Reading may lag behind WAL generation since it's "pulled" by the client and happens lazily after commit.
and where are this code is in the codebase?
src/backend/replication/logical/*
src/backend/replication/walsender.c
src/backend/access/transam/xlogreader.c
src/include/access/xlogreader.h
src/include/replication/output_plugin.h
contrib/test_decoding/
doc/src/sgml/logicaldecoding.sgml
On Mon, Jul 11, 2016 at 2:03 PM, Craig Ringer <craig@2ndquadrant.com> wrote: > On 9 July 2016 at 01:57, Joshua Bay <joshuabay93@gmail.com> wrote: >> and where are this code is in the codebase? > > src/backend/replication/logical/* > src/backend/replication/walsender.c > src/backend/access/transam/xlogreader.c > src/include/access/xlogreader.h > src/include/replication/output_plugin.h > contrib/test_decoding/ > doc/src/sgml/logicaldecoding.sgml Some other references: https://wiki.postgresql.org/images/7/77/Fosdem-2015-logical-decoding.pdf And some other examples of plugins: https://github.com/leptonix/decoding-json https://github.com/xstevens/decoderbufs https://github.com/confluentinc/bottledwater-pg (for kafka) https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw (wrote this one) -- Michael
On Mon, Jul 11, 2016 at 2:13 AM, Michael Paquier <michael.paquier@gmail.com> wrote:
>
> On Mon, Jul 11, 2016 at 2:03 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> > On 9 July 2016 at 01:57, Joshua Bay <joshuabay93@gmail.com> wrote:
> >> and where are this code is in the codebase?
> >
> > src/backend/replication/logical/*
> > src/backend/replication/walsender.c
> > src/backend/access/transam/xlogreader.c
> > src/include/access/xlogreader.h
> > src/include/replication/output_plugin.h
> > contrib/test_decoding/
> > doc/src/sgml/logicaldecoding.sgml
>
> Some other references:
> https://wiki.postgresql.org/images/7/77/Fosdem-2015-logical-decoding.pdf
>
> And some other examples of plugins:
> https://github.com/leptonix/decoding-json
> https://github.com/xstevens/decoderbufs
> https://github.com/confluentinc/bottledwater-pg (for kafka)
> https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw (wrote this one)
>
[1] https://github.com/eulerto/wal2json
--
Fabrízio de Royes Mello
Consultoria/Coaching PostgreSQL
>> Timbira: http://www.timbira.com.br
>> Blog: http://fabriziomello.github.io
>> Linkedin: http://br.linkedin.com/in/fabriziomello
>> Twitter: http://twitter.com/fabriziomello
>> Github: http://github.com/fabriziomello
Thanks a lot for your help!!
On Tue, Jul 12, 2016 at 4:42 PM, Fabrízio de Royes Mello <fabriziomello@gmail.com> wrote:
Regards,Nice, also we have wal2json [1].
On Mon, Jul 11, 2016 at 2:13 AM, Michael Paquier <michael.paquier@gmail.com> wrote:
>
> On Mon, Jul 11, 2016 at 2:03 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> > On 9 July 2016 at 01:57, Joshua Bay <joshuabay93@gmail.com> wrote:
> >> and where are this code is in the codebase?
> >
> > src/backend/replication/logical/*
> > src/backend/replication/walsender.c
> > src/backend/access/transam/xlogreader.c
> > src/include/access/xlogreader.h
> > src/include/replication/output_plugin.h
> > contrib/test_decoding/
> > doc/src/sgml/logicaldecoding.sgml
>
> Some other references:
> https://wiki.postgresql.org/images/7/77/Fosdem-2015-logical-decoding.pdf
>
> And some other examples of plugins:
> https://github.com/leptonix/decoding-json
> https://github.com/xstevens/decoderbufs
> https://github.com/confluentinc/bottledwater-pg (for kafka)
> https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw (wrote this one)
>
[1] https://github.com/eulerto/wal2json
--
Fabrízio de Royes Mello
Consultoria/Coaching PostgreSQL
>> Timbira: http://www.timbira.com.br
>> Blog: http://fabriziomello.github.io
>> Linkedin: http://br.linkedin.com/in/fabriziomello
>> Twitter: http://twitter.com/fabriziomello
>> Github: http://github.com/fabriziomello