Unit testing

Hi guys,

For the latest few weeks Neil and I have been discussing unit testing as
a means of testing Postgres more rigorously. A brief overview of the ideas
behind unit testing can be found here:
http://www.extremeprogramming.org/rules/unittests.html.

The basic idea is that you write small(ish) tests which test an individual
function. The tests encode the logical rules which the function should
follow. That is, for this particular input, a particular output.

Unit tests give us two things: 1) a lot more granularity than
regression tests, which may have a call path of many hundreds of
functions. We can pass functions atypical input, which they should be able
to deal with, but which they would not receive from the regression test
system. 2) For developers working on new functionality, unit tests will be
a complement to the regression test suite which allows them to quickly see
if their modifications to existing functions break.

Importantly, unit tests should have a minimal impact upon the source.
Tests for functions of a particular file can be placed in a seperate file.
We can create seperate unit test argument, like we do for regression
tests. The only problem I can think of is that many of the functions we
would want to test are static. We could just sed the static typing away,
however, using a rule that all static functions are defined with ^static.

There are frameworks which simplify the creation of test suites. These
frameworks provide a handful of functionality, generally: an assertion,
kind of like Assert() but if the assertion fails, it generates output
(unit test name, function being tested, some times more information) in a
standard format; a queueing system, so that individual tests can be
cobbled together into a suite; and, some provide the ability to run the
test in a child process so that segmentation violations can be detected
and so that a memory overrun doesn't stomp all over the actual test's
address space too. These frameworks usually come as a header and a C
language file can easily be compiled into the source to run the tests.

There are two frameworks I am aware of: cutest and Check.

Cutest, which you can find here: http://cutest.sourceforge.net/,  is a
very lightweight unit test suite. It provides and API to assert that
output, in various formats, matches a certain rule and it allows you to
queue tests into a suite. I think the API is a little more complex than
necessary. Where as with a standard Assert() in C we produce a test with a
boolean result, CuTest can do the tests itself (ie, if you want to assert
on a string comparison or an integer equality, CuTest provides functions
to actually do those operations). I think this is ugly and I guess we
could just use the standard boolean test.

It is licensed under a zlib style license which is, I believe, bsd compatible.

Check (http://check.sourceforge.net) is a library which the same
functionality as Cutest but also allows test designers to run the test in
a child process as well as have general setup and shutdown functions for a
range of tests. The latter is useful if setting up some state data for a
unit test is non-trivial (in terms of time) and would be required to run
for a sub set of the test suite. The actual API is much more like a C
Assert() than Cutest. It is LGPL.

We could of course write our own, we may give us some more freedom.

The point the project can greatly benefit from this kind of low level
testing. I'm happy to kick it off by working out the integration of a unit
test framework or developing a frame work which suites us -- depending on
what others think. I think I'll also start off with some low hanging fruit
by testing non-trivial functions in utils/adt/.

Neil and I will also be developing unit tests for the stored procedure
code we're working on for 8.1.

Ideas, comments, criticisms?

Thanks,

Gavin