Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier

On 01/05/2014 09:13 PM, Michael Paquier wrote:
>
>
> On Mon, Jan 6, 2014 at 4:51 AM, Mark Dilger <markdilger@yahoo.com 
> <mailto:markdilger@yahoo.com>> wrote:
> > I am building a regression test system for replication and came across
> > this email thread.  I have gotten pretty far into my implementation, but
> > would be happy to make modifications if folks have improvements to
> > suggest.  If the community likes my design, or a modified version based
> > on your feedback, I'd be happy to submit a patch.
> Yeah, this would be nice to look at, core code definitely needs to 
> have some more infrastructure for such a test suite. I didn't get the 
> time to go back to it since I began this thread though :)
>
> > Currently I am canibalizing src/test/pg_regress.c, but that could 
> instead
> > be copied to src/test/pg_regress_replication.c or whatever.  The 
> regression
> > test creates and configures multiple database clusters, sets up the
> > replication configuration for them, runs them each in nonprivileged mode
> > and bound to different ports, feeds all the existing 141 regression 
> tests
> > into the master database with the usual checking that all the right 
> results
> > are obtained, and then checks that the standbys have the expected
> > data.  This is possible all on one system because the database clusters
> > are chroot'ed to see their own /data directory and not the /data 
> directory
> > of the other chroot'ed clusters, although the rest of the system, 
> like /bin
> > and /etc and /dev are all bind mounted and visible to each cluster.
> Having vanilla regressions run in a cluster with multiple nodes and 
> check the results on a standby is the top of the iceberg though. What 
> I had in mind when I began this thread was to have more than a 
> copy/paste of pg_regress, but an infrastructure that people could use 
> to create and customize tests by having an additional control layer on 
> the cluster itself. For example, testing replication is not only a 
> matter of creating and setting up the nodes, but you might want to be 
> able to initialize, add, remove nodes during the tests. Node addition 
> would be either a new fresh master (this would be damn useful for a 
> test suite for logical replication I think), or a slave node with 
> custom recovery parameters to test replication, as well as PITR, 
> archiving, etc. Then you need to be able to run SQL commands on top of 
> that to check if the results are consistent with what you want.
>

I'd encourage anyone looking at implementing a testing suite for 
replication to look at the stuff we did for Slony at least to get some 
ideas.

We wrote a test driver framework (clustertest - 
https://github.com/clustertest/clustertest-framework) then some 
Javascript base classes for common types of operations.  An individual 
test is then written in Javascript that invokes methods either in the 
framework or base-class to do most of the interesting work.

http://git.postgresql.org/gitweb/?p=slony1-engine.git;a=blob;f=clustertest/disorder/tests/EmptySet.js;h=7b4850c1d24036067f5a659b990c7f05415ed967;hb=HEAD

as an example




> A possible input for a test that users could provide would be 
> something like that:
> # Node information for tests
> nodes
> {
>     {node1, postgresql.conf params, recovery.conf params}
>     {node2, postgresql.conf params, recovery.conf params, slave of node1}
> }
> # Run test
> init node1
> run_sql node1 file1.sql
> # Check output
> init node2
> run_sql node2 file2.sql
> # Check that results are fine
> # Process
>
> The main problem is actually how to do that. Having some smart shell 
> infrastructure would be simple and would facilitate (?) the 
> maintenance of code used to run the tests. On the contrary having a C 
> program would make the maintenance of code to run the tests more 
> difficult (?) for a trade with more readable test suite input like the 
> one I wrote above. This might also make the test input more readable 
> for a human eye, in the shape of what is already available in 
> src/test/isolation.
>
> Another possibility could be also to integrate directly a 
> recovery/backup manager in PG core, and have some tests for it, or 
> even include those tests directly with pg_basebackup or an upper layer 
> of it.
>
> > There of course is room to add as many replication tests as you like,
> > and the main 141 tests fed into the master could be extended to feed
> > more data and such.
> >
> > The main drawbacks that I don't care for are:
> >
> > 1) 'make check' becomes 'sudo make check' because it needs permission
> > to run chroot.
> -1 for that developers should not need to use root to run regression 
> suite.
>
> > 2) I have no win32 version of the logic
> For a first shot I am not sure that it matters much.
>
> > The main advantages that I like about this design are:
> >
> > 1) Only one system is required.  The developer does not need network
> > access to a second replication system.  Moreover, multiple database
> > clusters can be established with interesting replication hierarchies 
> between
> > them, and the cost of each additional cluster is just another chroot
> > environment
> An assumption of the test suite is I think to allow developers to 
> check for bugs on a local server only. This facilitates how the test 
> suite is written and you don't need to enter in things like VM 
> settings or cross-environment tests, things that could be done already 
> nicely by frameworks of the type Jenkins. What I think people would 
> like to have is that:
> cd src/test/replication && make check/installcheck
> And have the test run for them.
>
> Regards,
> -- 
> Michael