Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
От | Dilip Kumar |
---|---|
Тема | Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions |
Дата | |
Msg-id | CAFiTN-s6C0Yw9+k4FE7v7O_9iZ1+XTccOaz0Y9o-xrppcB45JA@mail.gmail.com обсуждение исходный текст |
Ответ на | Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions (Amit Kapila <amit.kapila16@gmail.com>) |
Ответы |
Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
(Amit Kapila <amit.kapila16@gmail.com>)
|
Список | pgsql-hackers |
On Thu, Jul 16, 2020 at 4:25 PM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Thu, Jul 16, 2020 at 12:23 PM Dilip Kumar <dilipbalaut@gmail.com> wrote: > > > > On Wed, Jul 15, 2020 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote: > > > > > > > > > Let me know what you think of the changes? > > > > I have reviewed the changes and looks fine to me. > > > > Thanks, I am planning to start committing a few of the infrastructure > patches (especially first two) by early next week as we have resolved > all the open issues and done an extensive review of the entire > patch-set. In the attached version, there is a slight change in one > of the commit messages as compared to the previous version. I would > like to describe in brief the first two patches for the sake of > convenience. Let me know if you or anyone else sees any problems with > these. > > The first patch in the series allows us to WAL-log subtransaction and > top-level XID association. The logical decoding infrastructure needs > to know which top-level > transaction the subxact belongs to, in order to decode all the > changes. Until now that might be delayed until commit, due to the > caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring > incremental decoding. So we also write the assignment info into WAL > immediately, as part of the next WAL record (to minimize overhead) > only when *wal_level=logical*. We can not remove the existing > XLOG_XACT_ASSIGNMENT WAL as that is required for avoiding overflow in > the hot standby snapshot. > > The second patch writes WAL for invalidations at command end with > wal_level=logical. When wal_level=logical, write invalidations at > command end into WAL so that decoding can use this information. This > patch is required to allow the streaming of in-progress transactions > in logical decoding. We still add the invalidations to the cache and > write them to WAL at commit time in RecordTransactionCommit(). This > uses the existing XLOG_INVALIDATIONS xlog record type, from the > RM_STANDBY_ID resource manager (see LogStandbyInvalidations for > details). So existing code relying on those invalidations (e.g. redo) > does not need to be changed. The invalidations written at command end > uses a new xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID > resource manager. See LogLogicalInvalidations for details. These new > xlog records are ignored by existing redo procedures, which still rely > on the invalidations written to commit records. The invalidations are > decoded and accumulated in top-transaction, and then executed during > replay. This obviates the need to decode the invalidations as part of > a commit record. > > The performance testing has shown that there is no performance penalty > with either of the patches but there is some additional WAL which in > most cases is 2-5% but in worst cases and for some specific DDL's it > is up to 15% with the second patch, however, that happens at > wal_level=logical only. We have considered an alternative to blow up > all caches on any DDL in WALSenders and that will have both CPU and > network overhead. For detailed results and analysis see [1][2]. > > [1] - https://www.postgresql.org/message-id/CAKYtNAqWkPpPFrdEbpPrCan3G_QAcankZarRKKd7cj6vQigM7w%40mail.gmail.com > [2] - https://www.postgresql.org/message-id/CAA4eK1L3PoiBw6uogB7jD5rmdT-GmEF4kOEccS1AWKuBhSkQkQ%40mail.gmail.com > The patch set required to rebase after committing the binary format option support in the create subscription command. I have rebased the patch set on the latest head and also added a test case to test streaming in binary format. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com
Вложения
В списке pgsql-hackers по дате отправления:
Следующее
От: "tsunakawa.takay@fujitsu.com"Дата:
Сообщение: RE: Transactions involving multiple postgres foreign servers, take 2