Обсуждение: pgsql: Add support for prepared transactions to built-in logical replic
Add support for prepared transactions to built-in logical replication. To add support for streaming transactions at prepare time into the built-in logical replication, we need to do the following things: * Modify the output plugin (pgoutput) to implement the new two-phase API callbacks, by leveraging the extended replication protocol. * Modify the replication apply worker, to properly handle two-phase transactions by replaying them on prepare. * Add a new SUBSCRIPTION option "two_phase" to allow users to enable two-phase transactions. We enable the two_phase once the initial data sync is over. We however must explicitly disable replication of two-phase transactions during replication slot creation, even if the plugin supports it. We don't need to replicate the changes accumulated during this phase, and moreover, we don't have a replication connection open so we don't know where to send the data anyway. The streaming option is not allowed with this new two_phase option. This can be done as a separate patch. We don't allow to toggle two_phase option of a subscription because it can lead to an inconsistent replica. For the same reason, we don't allow to refresh the publication once the two_phase is enabled for a subscription unless copy_data option is false. Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow Tested-By: Haiying Tang Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/a8fd13cab0ba815e9925dc9676e6309f699b5f72 Modified Files -------------- contrib/test_decoding/test_decoding.c | 12 +- doc/src/sgml/catalogs.sgml | 12 + doc/src/sgml/protocol.sgml | 291 ++++++++++++++++- doc/src/sgml/ref/alter_subscription.sgml | 5 + doc/src/sgml/ref/create_subscription.sgml | 37 +++ doc/src/sgml/ref/pg_dump.sgml | 7 +- src/backend/access/transam/twophase.c | 68 ++++ src/backend/catalog/pg_subscription.c | 34 ++ src/backend/catalog/system_views.sql | 2 +- src/backend/commands/subscriptioncmds.c | 131 +++++++- .../libpqwalreceiver/libpqwalreceiver.c | 10 +- src/backend/replication/logical/decode.c | 11 +- src/backend/replication/logical/logical.c | 31 +- src/backend/replication/logical/origin.c | 7 +- src/backend/replication/logical/proto.c | 217 ++++++++++++- src/backend/replication/logical/reorderbuffer.c | 25 +- src/backend/replication/logical/snapbuild.c | 33 +- src/backend/replication/logical/tablesync.c | 197 +++++++++-- src/backend/replication/logical/worker.c | 347 +++++++++++++++++++- src/backend/replication/pgoutput/pgoutput.c | 201 +++++++++--- src/backend/replication/slot.c | 1 + src/backend/replication/walreceiver.c | 2 +- src/bin/pg_dump/pg_dump.c | 20 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/psql/describe.c | 8 +- src/bin/psql/tab-complete.c | 2 +- src/include/access/twophase.h | 2 + src/include/catalog/catversion.h | 2 +- src/include/catalog/pg_subscription.h | 11 + src/include/catalog/pg_subscription_rel.h | 1 + src/include/replication/logical.h | 10 + src/include/replication/logicalproto.h | 73 ++++- src/include/replication/pgoutput.h | 1 + src/include/replication/reorderbuffer.h | 8 +- src/include/replication/slot.h | 7 +- src/include/replication/snapbuild.h | 5 +- src/include/replication/walreceiver.h | 7 +- src/include/replication/worker_internal.h | 3 + src/test/regress/expected/subscription.out | 109 ++++--- src/test/regress/sql/subscription.sql | 25 ++ src/test/subscription/t/021_twophase.pl | 359 +++++++++++++++++++++ src/test/subscription/t/022_twophase_cascade.pl | 235 ++++++++++++++ src/tools/pgindent/typedefs.list | 3 + 43 files changed, 2382 insertions(+), 191 deletions(-)
Re: pgsql: Add support for prepared transactions to built-in logical replic
От
Michael Paquier
Дата:
Hi Amit, On Wed, Jul 14, 2021 at 02:41:43AM +0000, Amit Kapila wrote: > Add support for prepared transactions to built-in logical replication. 021_twophase.pl, that has been introdeuced in this commit, is proving to be a bit unstable: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=idiacanthus&dt=2021-07-30%2009%3A56%3A04 # Failed test 'transaction is prepared on subscriber' # at t/021_twophase.pl line 324. # got: '1' # expected: '2' # Looks like you failed 1 test of 24. [12:14:02] t/021_twophase.pl .................. Thanks, -- Michael
Вложения
On Fri, Jul 30, 2021 at 6:17 PM Michael Paquier <michael@paquier.xyz> wrote: > > Hi Amit, > > On Wed, Jul 14, 2021 at 02:41:43AM +0000, Amit Kapila wrote: > > Add support for prepared transactions to built-in logical replication. > > 021_twophase.pl, that has been introdeuced in this commit, is proving > to be a bit unstable: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=idiacanthus&dt=2021-07-30%2009%3A56%3A04 > # Failed test 'transaction is prepared on subscriber' > # at t/021_twophase.pl line 324. > # got: '1' > # expected: '2' > # Looks like you failed 1 test of 24. > [12:14:02] t/021_twophase.pl .................. > Thanks for the report. I have shared my initial analysis in the corresponding -hackers thread [1]. [1] - https://www.postgresql.org/message-id/CAA4eK1%2B_0iNQ8Z%3DKVTjmmAqNX-hyv%2B1%2BfnZ-Yx8CVP%3DuAcekqw%40mail.gmail.com -- With Regards, Amit Kapila.
On Sat, Jul 31, 2021 at 10:11 AM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Fri, Jul 30, 2021 at 6:17 PM Michael Paquier <michael@paquier.xyz> wrote: > > > > Hi Amit, > > > > On Wed, Jul 14, 2021 at 02:41:43AM +0000, Amit Kapila wrote: > > > Add support for prepared transactions to built-in logical replication. > > > > 021_twophase.pl, that has been introdeuced in this commit, is proving > > to be a bit unstable: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=idiacanthus&dt=2021-07-30%2009%3A56%3A04 > > # Failed test 'transaction is prepared on subscriber' > > # at t/021_twophase.pl line 324. > > # got: '1' > > # expected: '2' > > # Looks like you failed 1 test of 24. > > [12:14:02] t/021_twophase.pl .................. > > > > Thanks for the report. I have shared my initial analysis in the > corresponding -hackers thread [1]. > I have pushed the fix for this [1]. [1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=eaf5321c352478266cebe2aa50ea4c34a8fdd2c7 -- With Regards, Amit Kapila.