Hot Standby and VACUUM FULL

The last item on my list before close is making VACUUM FULL and Hot
Standby play nicely together.

The options to do this were and still are:

(1) Add WAL messages for non-transactional relcache invalidations
(2) Allow system relations to be cluster-ed/vacuum full-ed.

(1) was how we did it originally and I believe it worked without
problem. We saw the opportunity to do (2) and it has been worth
exploring.

My approach to (2) was to look at this design-wise. Given my experience
with other aspects of relcache and invalidation, it all looked do-able
without problem. The design seems straightforward in a few ways, though
had a few special cases.

I attach a WIP patch that is sufficient to do psql -c "VACUUM FULL
pg_am;" successfully, it being the easiest case out of the special
cases. It's also easy to remove the special case-avoidance code in
vacuum.c to see the various failures that occur without further work.
I've added tests to cover the new cases, which currently cause make
check to fail solely because I haven't updated the test output, since
this is a WIP.

I'm not aware of any specific technical blockers to continuing with (2).

Having said that I now realise a few things I didn't before:

* Approach (2) effects the core of Postgres, even if you don't use Hot
Standby.
* I've had to remove 7 sanity checks to get the first few VACUUMs
working. ISTM that removing various basic checks in the code is not a
good thing.
* There are are more special cases than I realised at first: temp,
shared, with-toast, nailed, shared-and-nailed, pg_class, normal system.

Taken together, ISTM that the benefits of progressing towards (2) are
not worth the potential burst radius of any problems introduced. With
the larger number of special cases and the removal of checks we may be
less able to spot or cope with failure. Given that people are getting
edgy about code instability, I would say there's likely to be some here.

My feeling is that we should return to approach (1) for Postgres 9.0,
since we have an approach that works and isolates any change to just Hot
Standby related codepaths. I'm not personally keen to introduce core
changes unrelated to the new feature (HS).

Your thoughts, please.

--
 Simon Riggs           www.2ndQuadrant.com