Re: RFC: adding pytest as a supported test framework

Hi,

On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:
> On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote:
> > Just for context for the rest the email: I think we desperately need to move
> > off perl for tests. The infrastructure around our testing is basically
> > unmaintained and just about nobody that started doing dev stuff in the last 10
> > years learned perl.
> 
> Okay. Personally, I'm going to try to stay out of discussions around
> subtracting Perl and focus on adding Python, for a bunch of different
> reasons:

I think I might have formulated my paragraph above badly - I didn't mean that
we should move away from perl tests tomorrow, but that we need a path forward
that allows folks to write tests without perl.


> - Tests aren't cheap, but in my experience, the maintenance-cost math
> for tests is a lot different than the math for implementations.

At the moment they tend to be *more* expensive often, due to spurious
failures. That's mostly not perl's fault, don't get me wrong, but us not
having better infrastructure for testing complicated behaviour and/or testing
things on a more narrow basis.


> > > Problem 1 (rerun failing tests): One architectural roadblock to this
> > > in our Test::More suite is that tests depend on setup that's done by
> > > previous tests. pytest allows you to declare each test's setup
> > > requirements via pytest fixtures, letting the test runner build up the
> > > world exactly as it needs to be for a single isolated test. These
> > > fixtures may be given a "scope" so that multiple tests may share the
> > > same setup for performance or other reasons.
> >
> > OTOH, that's quite likely to increase overall test times very
> > significantly. Yes, sometimes that can be avoided with careful use of various
> > features, but often that's hard, and IME is rarely done rigiorously.
> 
> Well, scopes are pretty front and center when you start building
> pytest fixtures, and the complicated longer setups will hopefully
> converge correctly early on and be reused everywhere else. I imagine
> no one wants to build cluster setup from scratch.

One (the?) prime source of state in our tap tests is the
database. Realistically we can't just tear that one down and reset it between
tests without causing the test times to explode. So we'll have to live with
some persistent state.


> On a slight tangent, is this not a problem today?

It is, but that doesn't mean making it even bigger is unproblematic :)




> > I think part of the problem is that the information about what precisely
> > failed is often much harder to collect when testing multiple servers
> > interacting than when doing localized unit tests.
> >
> > I think we ought to invest a bunch in improving that, I'd hope that a lot of
> > that work would be largely independent of the language the tests are written
> > in.
> 
> We do a lot more acceptance testing than internal testing, which came
> up as a major complaint from me and others during the unconference.
> One of the reasons people avoid writing internal tests in Perl is
> because it's very painful to find a rhythm with Test::More.

What definition of internal tests are you using here?

I think a lot of our tests are complicated, fragile and slow because we almost
exclusively do end-to-end tests, because with a few exceptions we don't have a
way to exercise code in a more granular way.


> > > When it comes to third-party packages, which I think we're
> > > probably going to want in moderation, we would still need to discuss
> > > supply chain safety. Python is not as mature here as, say, Go.
> >
> > What external dependencies are you imagining?
> 
> The OAuth pytest suite makes extensive use of
> - psycopg, to easily drive libpq;

That's probably not going to fly. It introduces painful circular dependencies
between building postgres (for libpq), building psycopg (requiring libpq) and
testing postgres (requiring psycopg).

You *can* solve such issues, but we've debated that in the past, and I doubt
we'll find agreement on the awkwardness it introduces.


> - construct, for on-the-wire packet representations and manipulation; and

That seems fairly minimal.


> - pyca/cryptography, for easy generation of certificates and manual
> crypto testing.

That's a bit more painful, but I guess maybe not unrealistic?


> I'd imagine each would need considerable discussion, if there is
> interest in doing the same things that I do with them.

One thing worth thinking about is that such dependencies have to work on a
relatively large number of platforms / architectures. A lot of projects
don't...

Greetings,

Andres Freund