Re: Multi-xacts and our process problem

On Mon, May 11, 2015 at 05:33:04PM -0400, Bruce Momjian wrote:
> On Tue, May 12, 2015 at 12:29:56AM +0300, Heikki Linnakangas wrote:
> > On 05/12/2015 12:00 AM, Bruce Momjian wrote:
> > >Multi-xacts were made durable in Postgres 9.3 (released 2013-09-09) to
> > >allow primary-key-column-only locks.  1.7 years later, we are still
> > >dealing with bugs related to this feature.  Obviously, something is
> > >wrong.
> > >
> > >There were many 9.3 minor releases containing multi-xacts fixes, and
> > >these fixes have extended into 9.4.  After the first few bug-fix
> > >releases, I questioned whether we needed to revert or rework the
> > >feature, but got no positive response.  Only in the past few weeks have
> > >we got additional people involved.
> > 
> > The "revert or rework" ship had already sailed at that point. I
> 
> True.
> 
> > don't think we had much choice than just soldier through the bugs
> > after the release.
> 
> The problem is we "soldiered on" without adding any resources to the
> problem or doing a systematic review once it became clear one was
> necessary.

In both this latest emergency and the Nov-Dec 2013 one, several people showed
up and helped.  We did well in that respect, but your idea that we should have
started a systematic post-commit review is a good one.  Hoping other parts of
the change were fine, the first team dispersed.


There's a lot we might try, but I'll focus on just a couple of points.

The adversarial relationship between author and committer is an important
guarantor of quality.  For the toughest patches, the principal author should
seek out an independent committer rather than self-commit the patch.

When I want to keep a soon-to-be-committed patch's bugs out of PostgreSQL, I
can review it to find as many bugs as possible, or I can express nonspecific
mistrust.  That first option is expensive; if I did full-time patch review, I
might cover 1/4 of the changes.  That second option boils down to me telling a
committer that he is practicing bad judgment, which is painful for both of us
and unlikely to modify the patch's fate.  At the PgCon 2014 Developer Meeting,
it came out that most people had identified fklocks as the highest-risk 9.3
patch.  Here's an idea.  Shortly after the 9.5 release notes draft, let's take
a secret ballot to identify the changes threatening the most damage through
undiscovered bugs.  (Let's say the electorate consists of every committer and
every person who reviewed at least one patch during the release cycle.)
Publish the three top vote totals.  This serves a few purposes.  It sends a
message to each original committer that the community doubts his handling of
the change.  The secret ballot helps voters be honest, and seven votes against
your commit is hard to ignore.  It's a hint to that committer to drum up more
reviews and testing, to pick a simpler project next time, or even to revert.
The poll results would also help target beta testing and post-commit reviews.
For example, I would plan to complete a full post-commit review of one patch
in the list.

Thanks,
nm