truncating pg_multixact/members

I started looking at bug #8673 some days ago, and I identified three
separate issues that need fixing:

1. slru.c doesn't consider file names longer than 4 hexadecimal chars.

2. pg_multixact/members truncation requires more intelligence to avoid
removing files that are still needed.  Right now we use modulo-2^32
arithmetic, but this doesn't work because the useful range can span
longer than what we can keep within that range.

3. New pg_multixact/members generation requires more intelligence to
avoid stomping on files from the previous wraparound cycle.  Right now
there is no defense against this at all.


Fixing (1) is simple: we can have each SLRU user declare how many digits
to have in file names.  All existing users but pg_multixact/members
should declare 4 digits; that one should declare 5.  That way, the
correct number of zeroes are allocated at the start point and we get
nice, equal-width file names.  Eventually, predicate.c can change to
wider file names and get rid of some strange code it has to deal with
overrun.

For 9.3, I propose we skip this and tweak the code to consider files
whose names are 4 or 5 chars in length, to remain compatible with
existing installations that have pg_multixact/member having a mixture of
4-char and 5-char file names.


For (2) a simple-minded proposal is to have a new SlruScanDirectory
callback that knows to delete only files within a certain range.  Then,
at truncate time, collect the existing valid range (i.e. files
containing multixacts between oldestMulti and nextMulti) and delete
files outside this range.  However there is a race condition: if
pg_multixact/member grows concurrently while the truncation is
happening, the new files would be outside the range and would be
deleted.  There is no bound to how much the directory can grow, so it
doesn't seem reasonable to just add some arbitrary safety limit.  I see
three possible fixes:

#2a Interlock directory truncation with new file generation: in
GetNewMultiXactId and TruncateMultiXact grab a lock (perhaps a boolean
in MultiXactState, or perhaps just a new LWLock) to exclude each from
the other.  That way, truncation can obtain a range that will continue
to be meaningful until truncation is complete, and no new files will be
erased.

#2b During truncation, first obtain a directory listing, *then* compute
the range of files to keep, then delete files outside that range but
only if they are present in the listing previously obtained.  That way,
files created during truncation are not removed.

#2c At start of truncation, save end-of-range in MultiXactState.  This
state is updated by GetNewMultiXactId as new files are created.  That
way, before each new file is created, the truncation routine knows to
skip it.

I don't like #2a because of concurrency loss, and I don't like #2b
because it seems ugly and potentially slow.  #2c seems to most
reasonable way to attack this problem, but if somebody has a differing
opinion please voice it.


For (3) there is a hand-wavy idea that we can compare the oldest offset
to the next offset, and avoid enlarging (raise an ERROR) if an overrun
occurs; but we don't have the oldest offset stored anywhere convenient.
We would have to scan multixact/offsets at start of service to determine
it, and then perhaps keep it in MultiXactState.  Arguably this info
should be part of pg_control, but we don't have that in 9.3 so we'll
have to find some other idea.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services