C testing for Postgres

During the unconference at PGCon in Ottawa, I asked about writing C-based tests for Postgres. There was interest in trying a tool and also some hesitation to depend on a third-party library. So, I wrote a tool that I'd like to contribute to Postgres. I’ve been calling it cexpect [1].

cexpect is a general-use library for creating test suites in C. It includes:

- a core library for creating and running suites of tests.

- a standard set of test expectations for C

- a default formatter (dots)

- an extensible matcher framework

- an extensible formatter framework

Why add a testing framework for C to Postgres? 

An xUnit-style test framework [2] is a tool for writing tests that is not currently an option for people hacking on Postgres. 

• C-based tests could help increase code coverage in parts of the codebase that are difficult to reach with a regress-style test (for example: gin posting list compression [3]). 

• Writing tests for internal components help developers become more familiar with a codebase. 

• Writing C-based tests go beyond providing regression value by providing feedback on design decisions like modularity and dependencies. 

• Test suites already existing in `src/test/modules` could benefit from having a consistent way to declare expectations, run suites, and display results.

• Forks and extensions that write tests could benefit from a testing framework provided by the upstream project. An extensible framework will allow forks and extensions to create their own matchers and formatters without changing the core framework.

The extensible matcher framework has benefits for decoupled, isolated unit tests, and also  high-level system tests. The complexity of a common assertions can be packaged into a matcher - abstracting the complexity and making the assertion easier to reuse.

For example, there could be expectation matchers specifically crafted for the domain of Postgres:

`expect(xlog, to_have_xlog_record(XLOG_XACT_PREPARE))`

`expect(“postgresadmin”, to_be_an_admin_user())`

The matchers that come with cexpect out of the box are for C datatypes:

`expect(1, is_int_equal_to(1))`

`expect(1 == 2, is_false())`

`expect(“some random string”, is_string_containing(“and”))’

… and more, with the goal of having a matcher for all standard data types.

The extensible formatter framework could be used to create a test-anything protocol (TAP) output, familiar to Postgres hackers. It could also be used to insert test results into a database table for later analysis, or create a consistent way of reporting results from a user-defined function - the pattern often used for creating tests under `src/test/modules`.

How does it work?

Create an executable that links to the core shared library, include the core library headers, create a suite, add some tests, and run it.

test.c:

```

#include "cexpect.h"

#include "cexpect_cmatchers.h"

void some_passing_test(Test *test) {

    expect(test, 1, is_int_equal_to(1));

}

int main(int argc, char *args[]) {

    Suite *suite = create_suite("Example test");

    add_test(suite, some_passing_test);

    start_cexpect(suite);

}

```

Running test.c:

```bash

export DYLD_LIBRARY_PATH=$path_to_cexpect_library

export compile_flags=”-Wno-int-conversion -Wno-pointer-to-int-cast test.c -L $path_to_cexpect_library -I $path_to_cexpect_headers”

gcc $compile_flags -l cexpect -o test.o

./test.o

Running suite: Example test

.

Summary:

Ran 1 test(s).

1 passed, 0 failed, 0 pending

```

Rather than post a patch, I'd rather start a conversation first. I'm guessing there are some improvements that we'd want to make (for example: the Makefile) before commiting a patch. Let's iterate on improvements before creating a formal patch.

Thoughts?

Thanks,

Adam Berlin

Software Engineer at Pivotal Greenplum

[1] https://github.com/berlin-ab/cexpect

[2] https://en.wikipedia.org/wiki/XUnit

[3] https://coverage.postgresql.org/src/backend/access/gin/ginpostinglist.c.gcov.html