Re: RFC: adding pytest as a supported test framework

On Wed, Jun 12, 2024 at 8:50 AM Andres Freund <andres@anarazel.de> wrote:
> I think I might have formulated my paragraph above badly - I didn't mean that
> we should move away from perl tests tomorrow, but that we need a path forward
> that allows folks to write tests without perl.

Okay, agreed.

> > - Tests aren't cheap, but in my experience, the maintenance-cost math
> > for tests is a lot different than the math for implementations.
>
> At the moment they tend to be *more* expensive often, due to spurious
> failures. That's mostly not perl's fault, don't get me wrong, but us not
> having better infrastructure for testing complicated behaviour and/or testing
> things on a more narrow basis.

Well, okay, but I'm not sure how to respond to this in the frame of
this discussion. Bad tests will continue to exist. I am trying to add
a tool that, in my view, has made it easier for me to test complicated
behavior than what we currently have. I can't prove that it will solve
other issues too.

> > Well, scopes are pretty front and center when you start building
> > pytest fixtures, and the complicated longer setups will hopefully
> > converge correctly early on and be reused everywhere else. I imagine
> > no one wants to build cluster setup from scratch.
>
> One (the?) prime source of state in our tap tests is the
> database. Realistically we can't just tear that one down and reset it between
> tests without causing the test times to explode. So we'll have to live with
> some persistent state.

Yes? If I've given the impression that I disagree, sorry; I agree.

> > On a slight tangent, is this not a problem today?
>
> It is, but that doesn't mean making it even bigger is unproblematic :)

Given all that's been said, I don't understand why you think the
problem would get bigger. We would cache expensive state that we need,
including the cluster, and pytest lets us do that, and my test suite
does that. I've never written a suite that spun up a separate cluster
for every single test and then immediately threw it away.

(If you want to _enable_ that behavior, to test in extreme isolation,
then pytest lets you do that too. But it's not something we should do
by default.)

> > We do a lot more acceptance testing than internal testing, which came
> > up as a major complaint from me and others during the unconference.
> > One of the reasons people avoid writing internal tests in Perl is
> > because it's very painful to find a rhythm with Test::More.
>
> What definition of internal tests are you using here?

There's a spectrum from unit-testing unexported functions all the way
to end-to-end acceptance, and personally I believe that anything
finer-grained than end-to-end acceptance is unnecessarily painful. My
OAuth suite sits somewhere in the middle, where it mocks the protocol
layer and can test the client and server as independent pieces. Super
useful for OAuth, which is asymmetrical.

I'd like to additionally see better support for unit tests of backend
internals, but I don't know those seams as well as all of you do and I
should not be driving that. I don't think Python will necessarily help
you with it. But it sure helped me break apart the client and the
server while enjoying the testing process, and other people want to do
that too, so that's what I'm pushing for.

> I think a lot of our tests are complicated, fragile and slow because we almost
> exclusively do end-to-end tests, because with a few exceptions we don't have a
> way to exercise code in a more granular way.

Yep.

> That's probably not going to fly. It introduces painful circular dependencies
> between building postgres (for libpq), building psycopg (requiring libpq) and
> testing postgres (requiring psycopg).

I am trying very hard not to drag that, which I understand is
controversial and is in no way a linchpin of my proposal, into the
discussion of whether or not we should try supporting pytest.

I get it; I understand that the circular dependency is weird; there
are alternatives if it's unacceptable; none of that has anything to do
with Python+pytest.

> One thing worth thinking about is that such dependencies have to work on a
> relatively large number of platforms / architectures. A lot of projects
> don't...

Agreed.

Thanks,
--Jacob