Re: [HACKERS] Thoughts on unit testing?

On Mon, Aug 14, 2017 at 2:31 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 14 August 2017 at 03:19, Peter Geoghegan <pg@bowt.ie> wrote:
>> It is my understanding that much of the benefit of unit testing comes
>> from maintainability. It's something that goes well with design by
>> contract. Writing unit tests is a forcing function. It requires
>> testable units, making the units more composable. The programmer must
>> be very deliberate about state, and must delineate code as testable
>> units. Unit tests are often supposed to be minimal, to serve as a kind
>> of design document.
>
>
> This is why I personally only find true unit tests useful as part of large
> software packages like Pg when they cover functional units that can be
> fairly well isolated.
>
> However, I'm not sure that Thomas meant unit tests in the formal sense
> isolated and mocked, but rather in the sense of test procedures written in C
> to excercise portions of Pg's code that are not easily/conveniently tested
> from SQL.

I'm interested in both.  The former would be marvellous but difficult
to get to from here, and the latter seems quite achievable and very
useful.  I'm hoping we can start figuring out a good way to do the
latter and make use of it initially for isolated new code, and then
see where that leads.  It's notoriously difficult to retrofit unit
tests to large old code bases and I have a few scars from doing that
(or trying) with big proprietary C and C++ systems.  But I'm
interested in gradual changes that cutting down on global state that
makes it difficult.  Here you can see a tiny example of that:

https://www.postgresql.org/message-id/CAEepm%3D1vUNNBUvTfP%2BJ7wgSqtEbb5NAg01VoZ2hPVyKG2qo8Qw%40mail.gmail.com

Things like this, even after the massive effort of moving enough
things into it to make it useful, wouldn't necessarily allow for
so-called mock session creation (ie a different implementation --
which I guess would involve interacting with the session via a vtable
of function pointers, or some compile time trickery?), but it's a
small step, and might eventually help you set up and tear down test
environments.

> Pg lacks the strict module delineation needed to make formal unit testing
> practical, as you have noted. That doesn't mean test frameworks couldn't be
> useful. I routinely write tests for reasonably isolated functionality in my
> code by writing throwaway SQL-callable functions to exercise it and
> pg_regress tests to run them. It's not strict unit testing as the rest of
> Pg's APIs aren't mocked away, but it's very practical small-unit integration
> testing that helps catch issues.
>
> I wouldn't mind having an easier and nicer way to do that built in to Pg,
> but don't have many ideas about practical, low-maintenance ways to achieve
> it.

I see now that the idea I mentioned earlier amounts to doing what
you've been doing, except with more tooling and conventions to lower
the barrier to use it.  A set of standard test macros, automatic
discovery of your tests, collocation of tests alongside the module
under test in the source tree, good test result reporting etc.  I'm
planning to come back to this in a while and try out some ideas for
the minimum useful scaffolding, but I'd love to hear any suggestions.

-- 
Thomas Munro
http://www.enterprisedb.com