Re: Unit testing

Neil Conway wrote:

>[ apologies if this mail is poorly formatted, posted via webmail ]
>
>Gavin Sherry said:
>  
>
>>For the latest few weeks Neil and I have been discussing unit testing as
>>a means of testing Postgres more rigorously.
>>    
>>
>
>I should note that we've also been looking at some other ideas, including
>different approaches to testing, static analysis, and model checking.
>
>  
>
>>The only problem I can think of is that many of the functions we
>>would want to test are static. We could just sed the static typing away,
>>however, using a rule that all static functions are defined with ^static.
>>    
>>
>
>Another approach would be to have a "configure" flag to enable unit
>testing that would define "static" to nothing when enabled. It would be
>nice to have access to the prototypes of static functions while writing
>unit tests: we could either do without that, or have a script to generate
>the header files automatically.
>
>BTW, I think that unit testing could probably only be enabled when a
>configure flag is specified in any case: unfortunately the changes needed
>to implement it may be rather invasive.
>
>  
>
>>Where as with a standard Assert() in C we produce a test with a
>>boolean result, CuTest can do the tests itself (ie, if you want to assert
>>on a string comparison or an integer equality, CuTest provides functions
>>to actually do those operations). I think this is ugly and I guess we
>>could just use the standard boolean test.
>>    
>>
>
>I don't think it's ugly. FWIW, CuTest probably uses that approach because
>it is what most of the SUnit-derived testing frameworks do. You can always
>use CuAssert() if you want to write the rest of the assertion condition
>yourself.
>
>I think one challenge Gavin didn't mention is how easy (or not) it will be
>to write unit tests for deeply-internal parts of the backend: unit testing
>utils/adt and the like is all well and good, but there really isn't much
>point if that's all we can test. The problem with testing the guts of the
>backend is that a given backend function typically requires an enormous
>amount of state -- it will often make some pretty specific assumptions
>about the environment in which it is executing. I'm not sure if there is a
>simple way to solve this -- writing the first few deep-internals unit
>tests is probably going to be pretty painful. But I think there are a few
>reasons to be optimistic:
>
>- once we've written a few such tests, we can begin to see the
>initialization / setup code that is required by multiple tests, and
>refactor this out into separate functions in the backend. Eventually, the
>code that is invoked to do _real_ backend startup would be just another
>client of the same set of shared initialization functions that are used to
>initialize the environment for unit tests.
>
>- we don't need to write tests for the *entire* source tree before we
>begin to see some payback. Once we have a good test suite for a specific
>component (say, utils/adt or FE libpq), developers should be able to see
>the gains (and hassles) of unit testing; if people like it it should be
>easy to incrementally add more tests.
>
>One final note: it is a hassle to unit test a 300 line function because of
>all the different code paths and error conditions such a function usually
>has. I think a natural pattern will be to test small bits of functionality
>and refactor as you go: rather than trying to test a huge function, we
>ought to pull a distinct piece of functionality out and into its own
>function, which will be much easier to unit test by itself. So unit
>testing and refactoring the code to be divided into smaller, more granular
>functions tends to go hand in hand. Now, we can debate about whether the
>resulting functions are in good style (I strongly believe they are), but I
>thought I'd add that.
>
>
>  
>

a few thoughts.

1. Small functions are good. My personal rule of thumb is that if I 
can't read it all in one screenful it might be too big (and yes, I know 
I have broken that rule sometimes).
As time goes on, we seem to have a habit of adding bits and pieces of 
stuff inline rather than in a separate function. Short story: many years 
ago I was confronted with a 1000-line "if" statement with many levels, 
and was told that it was to avoid the overhead of function calls (on an 
architecture where the overhead was known to be very low, and where the 
compiler supported inlining anyway). It cost me days and days of work to 
find the right places to stuff my mods.

2. Won't dissolving away "static" cause naming conflicts?

3. Unit testing frameworks are best suited to component-based 
architectures, ISTM. I'm not sure that one would fit Postgres very well. 
Retrofitting unit testing is a lot harder than starting out doing it 
from day 1.

cheers

andrew