Re: logical changeset generation v6

On Thu, Sep 19, 2013 at 7:43 AM, Andres Freund <andres@2ndquadrant.com> wrote:
>> More generally, the thing that bugs me about this approach is that
>> logical replication is not really special, but what you've done here
>> MAKES it special. There are plenty of other situations where we are
>> too aggressive about holding back xmin.  A single-statement
>> transaction that selects from a single table holds back xmin for the
>> entire cluster, and that is Very Bad.  It would probably be unfair to
>> say, well, you have to solve that problem first.  But on the other
>> hand, how do we know that the approach you've adopted here isn't going
>> to make the more general problem harder to solve?  It seems invasive
>> at a pretty low level.

I agree that it's invasive, but I am doubtful that pegging the xmin in
a more granular fashion precludes this kind of optimization. We might
have to generalize what Andres has done, which could mean eventually
throwing it out and starting from scratch, but I have a hard time
seeing how that implies an appreciable cost above solving the general
problem first (now that Andres has already implemented the
RecentGlobalDataXmin thing). As I'm sure you appreciate, the cost of
doing the opposite - of solving the general problem first - may be
huge: waiting another release for logical changeset generation.

> The reason why I think it's actually different is that the user actually
> has control over how long transactions are running on the primary. They
> don't really control how fast a replication consumer consumes and how
> often it sends feedback messages.

Right. That's about what I said last year.

I find the following analogy useful: A logical changeset generation
implementation without RecentGlobalDataXmin is kind of like an
old-fashioned nuclear reactor, like the one they had at Chernobyl.
Engineers have to actively work in order to prevent it from
overheating. However, an implementation with RecentGlobalDataXmin is
like a modern, much safer nuclear reactor. Engineers have to actively
work to keep the reactor heated. Which is to say, with
RecentGlobalDataXmin a standby that dies cannot bloat the master too
much (almost as with hot_standby_feedback - that too requires active
participation from the standby to do harm to the master). Without
RecentGlobalDataXmin, the core system and the plugin at the very least
need to worry about that case when a standby dies.

I have a little bit of feedback that I forgot to mention in my earlier
reviews, because I thought it was too trivial then: something about
the name pg_receivellog annoys me in a way that the name
pg_receivexlog does not. Specifically, it looks like someone meant to
type pg_receivelog but fat-fingered it.

-- 
Peter Geoghegan