Обсуждение: Logical Decoding follows timelines
Currently, it doesn't. This patch is a WIP version of doing that, but only currently attempts to do this in the WALSender. Objective is to allow cascaded logical replication. Very WIP, but here for comments. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Вложения
On 12/15/2014 08:54 PM, Simon Riggs wrote: > Currently, it doesn't. > > This patch is a WIP version of doing that, but only currently attempts > to do this in the WALSender. > > Objective is to allow cascaded logical replication. > > Very WIP, but here for comments. With the patch, XLogSendLogical uses the same logic to calculate SendRqstPtr that XLogSendPhysical does. It would be good to refactor that into a common function, rather than copy-paste. SendRqstPtr isn't actually used for anything in XLogSendLogical. - Heikki
On 16 December 2014 at 14:25, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: > On 12/15/2014 08:54 PM, Simon Riggs wrote: >> >> Currently, it doesn't. >> >> This patch is a WIP version of doing that, but only currently attempts >> to do this in the WALSender. >> >> Objective is to allow cascaded logical replication. >> >> Very WIP, but here for comments. > > > With the patch, XLogSendLogical uses the same logic to calculate SendRqstPtr > that XLogSendPhysical does. It would be good to refactor that into a common > function, rather than copy-paste. Some of the logic is similar, but not all. > SendRqstPtr isn't actually used for anything in XLogSendLogical. It exists to allow the call which resets TLI. I'll see if I can make it exactly identical; I didn't think so when I first looked, will look again. Thanks -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 16 December 2014 at 21:17, Simon Riggs <simon@2ndquadrant.com> wrote: >>> This patch is a WIP version of doing that, but only currently attempts >> With the patch, XLogSendLogical uses the same logic to calculate SendRqstPtr >> that XLogSendPhysical does. It would be good to refactor that into a common >> function, rather than copy-paste. > > Some of the logic is similar, but not all. > >> SendRqstPtr isn't actually used for anything in XLogSendLogical. > > It exists to allow the call which resets TLI. > > I'll see if I can make it exactly identical; I didn't think so when I > first looked, will look again. Yes, that works. New version attached -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Вложения
On 12/17/2014 10:35 AM, Simon Riggs wrote: > On 16 December 2014 at 21:17, Simon Riggs <simon@2ndquadrant.com> wrote: > >>>> This patch is a WIP version of doing that, but only currently attempts > >>> With the patch, XLogSendLogical uses the same logic to calculate SendRqstPtr >>> that XLogSendPhysical does. It would be good to refactor that into a common >>> function, rather than copy-paste. >> >> Some of the logic is similar, but not all. >> >>> SendRqstPtr isn't actually used for anything in XLogSendLogical. >> >> It exists to allow the call which resets TLI. >> >> I'll see if I can make it exactly identical; I didn't think so when I >> first looked, will look again. > > Yes, that works. New version attached Some comments, mostly on readability (not all of these were this patch's fault): > /* > * Check that the timeline the client requested for exists, and > * the requested start location is on that timeline. > */ > (void) ReadSendTimeLine(cmd->timeline); > > /* > * Found the requested timeline in the history. Check that > * requested startpoint is on that timeline in our history. > * > * This is quite loose on purpose. We only check that we didn't > * fork off the requested timeline before the switchpoint. We > * don't check that we switched *to* it before the requested > * starting point. This is because the client can legitimately > * request to start replication from the beginning of the WAL > * segment that contains switchpoint, but on the new timeline, so > * that it doesn't end up with a partial segment. If you ask for a > * too old starting point, you'll get an error later when we fail > * to find the requested WAL segment in pg_xlog. > * > * XXX: we could be more strict here and only allow a startpoint > * that's older than the switchpoint, if it's still in the same > * WAL segment. > */ The first comment implies that the ReadSendTimeLine call checks that the requested start location is on the timeline, but that's actually done by the code that follows the second comment. I would merge these two comments, and move the ReadSendTimeLine call below the merged comment. > @@ -577,8 +571,8 @@ StartReplication(StartReplicationCmd *cmd) > * that's older than the switchpoint, if it's still in the same > * WAL segment. > */ > - if (!XLogRecPtrIsInvalid(switchpoint) && > - switchpoint < cmd->startpoint) > + if (!XLogRecPtrIsInvalid(sendTimeLineValidUpto) && > + sendTimeLineValidUpto < cmd->startpoint) > { > ereport(ERROR, > (errmsg("requested starting point %X/%X on timeline %u is not in this server's history", IMHO using the local 'switchpoint' variable was more clear. > @@ -941,6 +936,8 @@ StartLogicalReplication(StartReplicationCmd *cmd) > * Force a disconnect, so that the decoding code doesn't need to care > * about an eventual switch from running in recovery, to running in a > * normal environment. Client code is expected to handle reconnects. > + * This covers the race condition where we are promoted half way > + * through starting up. > */ > if (am_cascading_walsender && !RecoveryInProgress()) > { We could exit recovery immediately after this check. Why is this check needed? > /* > + * Find the timeline for the start location, or throw an error. > + * > + * Logical replication relies upon replication slots. Each slot has a > + * single timeline history baked into it, so this should be easy. > + */ I don't understand what "baked in" means here. > + /* > + * Get the SendRqstPtr and follow any timeline changes. > + */ > + SendRqstPtr = GetLatestRequestPtr(); The old comment used to say "Figure out how far we can safely send the WAL". I think that was much more clear. It's not clear what following timeline changes means here, and the fact that it "gets the SendRqstPtr" is obvious from the code. > + > +static XLogRecPtr > +GetLatestRequestPtr(void) This function desperately needs comment to explain what it does. I don't much like its name either. > +static TimeLineID > +ReadSendTimeLine(TimeLineID tli) Ditto. This function is also missing a "return". I think it would slightly more intuitive if this function didn't set the global variables directly, but simply returned the returned values to the caller. - Heikki
On 2015-01-03 12:07:29 +0200, Heikki Linnakangas wrote: > >@@ -941,6 +936,8 @@ StartLogicalReplication(StartReplicationCmd *cmd) > > * Force a disconnect, so that the decoding code doesn't need to care > > * about an eventual switch from running in recovery, to running in a > > * normal environment. Client code is expected to handle reconnects. > >+ * This covers the race condition where we are promoted half way > >+ * through starting up. > > */ > > if (am_cascading_walsender && !RecoveryInProgress()) > > { > > We could exit recovery immediately after this check. Why is this check > needed? I probably wrote that ched and I don't think it really is needed. I think that's a remnant of what the physical pendant used to do. I think this needs slightly more abstraction because the infrastructure is local to walsender.c - but logical decoding is also possible via SQL. I'm not yet sure how that should look like. It'd be awesome if in the course of that we could get rid of the nearly duplicated XLogRead() :( Simon, have you checked that this actually correctly follows timelines? Afaics the patch as is won't allow to start logical decoding on a standby. To allow logical decoding from clients I (apperently) wrote the the following comment:/* ---- * TODO: We got to change that someday soon... * * There's basically three things missing to allowthis: * 1) We need to be able to correctly and quickly identify the timeline a * LSN belongs to * 2) We need toforce hot_standby_feedback to be enabled at all times so * the primary cannot remove rows we need. * 3) support droppingreplication slots referring to a database, in * dbase_redo. There can't be any active ones due to HS recovery* conflicts, so that should be relatively easy. * ---- */if (RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("logical decoding cannot be used while in recovery"))); You're implementing 1) here. 3) doesn't look very challenging. But 2) imo is rather more interesting/complex. I guess we'd have to force that streaming replication is used, that a physical replication slot is used and that hot_standby_feedback is enabled. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
<div dir="ltr"><br /><div class="gmail_extra"><br /><div class="gmail_quote">On Wed, Dec 17, 2014 at 5:35 PM, Simon Riggs<span dir="ltr"><<a href="mailto:simon@2ndquadrant.com" target="_blank">simon@2ndquadrant.com</a>></span> wrote:<br/><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On16 December 2014 at 21:17, Simon Riggs <<a href="mailto:simon@2ndquadrant.com">simon@2ndquadrant.com</a>>wrote:<br /><br /> >>> This patch is a WIP versionof doing that, but only currently attempts<br /><br /></span><span class="">>> With the patch, XLogSendLogicaluses the same logic to calculate SendRqstPtr<br /> >> that XLogSendPhysical does. It would be good torefactor that into a common<br /> >> function, rather than copy-paste.<br /> ><br /> > Some of the logic issimilar, but not all.<br /> ><br /> >> SendRqstPtr isn't actually used for anything in XLogSendLogical.<br />><br /> > It exists to allow the call which resets TLI.<br /> ><br /> > I'll see if I can make it exactly identical;I didn't think so when I<br /> > first looked, will look again.<br /><br /></span>Yes, that works. New versionattached<br /></blockquote></div><br /></div><div class="gmail_extra">Moved patch to CF 2015-02 to not lose trackof it, also because it does not seem it received a proper review.<br />-- <br /><div class="gmail_signature">Michael<br/></div></div></div>
On Fri, Feb 13, 2015 at 4:57 PM, Michael Paquier wrote: > Moved patch to CF 2015-02 to not lose track of it, also because it does not > seem it received a proper review. This patch does not apply anymore, so attached is a rebased version. The comments mentioned here have not been addressed: http://www.postgresql.org/message-id/54A7BF61.9080708@vmware.com Also, what kind of tests have been done? Logical decoding cannot be used while a node is in recovery. Regards, -- Michael
Вложения
On 21 April 2015 at 05:49, Michael Paquier <michael.paquier@gmail.com> wrote:
--
On Fri, Feb 13, 2015 at 4:57 PM, Michael Paquier wrote:
> Moved patch to CF 2015-02 to not lose track of it, also because it does not
> seem it received a proper review.
This patch does not apply anymore, so attached is a rebased version.
The comments mentioned here have not been addressed:
http://www.postgresql.org/message-id/54A7BF61.9080708@vmware.com
Also, what kind of tests have been done? Logical decoding cannot be
used while a node is in recovery.
Returned with Feedback, I think. I have a new approach to be coded for next release.
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services