Recovery Test Framework

Recovery doesn't have a test framework as yet. I would like to add one
for this release, especially since we have so much recovery-related code
being added to the release (and manual testing is so time consuming).
Testing Hot Standby will also test sync rep, PITR etc, and could easily
uncover a few problems hiding in the background that have lain dormant.

The current regression tests are all self-contained tests that create
objects, insert data, run tests and then cleanup again. Almost every
single test case is read-write.

This gives a few problems for recovery & Hot Standby
* tests cannot easily be split so that read/write happens on master and
test execution happens on standby (or on both master and standby)
* there is no easy way to synchronise object creation on master and test
execution on standby

So I propose to setup two test schedules
* rep_master_schedule
* rep_standby_schedule
to be executed using pg_regress concurrently on separate database
servers running on different ports on the same system.

A test table would keep track of which tables have had their
prerequisites met, and rep_standby_schedule would wait until a test was
correctly set up before running the test. This would be achieved using
the attached test framework code.

We would then include newly written tests, rather than attempt to use
existing tests - but use the same framework of /sql /out /expected. Some
of these have already been written for HS.

Is this something the community would like to see included within the
distribution, or should we just keep or private and publish test results
using it. I would prefer the former since it would allow us to prove
that the code works and to be able to check for specific regressions as
bugs appear. It may also help the community to work together on the
various aspects of recovery code that are being included in 8.4.

It would be massively cool to be able to add this to the build farm.
There would be few blockers because with two servers running on same
system there are few OS specific aspects to this.

If people can discuss what would be required we should be able to get it
done in the near term, i.e. over next 2-3 weeks.

Comments?

--
 Simon Riggs           www.2ndQuadrant.com
 PostgreSQL Training, Services and Support