Re: RFC: adding pytest as a supported test framework

On Thu, 13 Jun 2024 at 15:38, Robert Haas <robertmhaas@gmail.com> wrote:
> For whatever reason, it seems like every piece of
> infrastructure that the PostgreSQL community has suffers from severe
> neglect. Literally everything I know of either has one or maybe two
> very senior hackers maintaining it, or no maintainer at all. Andrew
> maintains the buildfarm and it evolves quite slowly. Andres did all
> the work on meson, with some help from Peter. Thomas maintains cfbot
> as a skunkworks. The Perl-based TAP test framework gets barely any
> love at all. The CommitFest application is pretty much totally
> stagnant, and in fact is a great example of what I'm talking about
> here: I wrote an original version in Perl and somebody -- I think
> Magnus -- rewrote it in a more maintainable framework -- and then the
> development pace went to basically zero. All of this stuff is critical
> project infrastructure and yet it feels like nobody wants to work on
> it.

Overall, I agree with the sentiment of us not maintaining our tooling
well (although I think meson maintenance has been pretty decent so
far). I think there's a bunch of reasons for this (not all apply to
each of the tools):
1. pretty much solely maintained by senior community members who don't
have time to maintain it
2. no clear way to contribute. e.g. where should I send a patch/PR for
the commitfest app, or the cfbot?
3. (related to 1) unresponsive when somehow contributions are actually
sent in (I have two open PRs on the cfbot app from 3 years ago without
any response)

I think 1 & 3 could be addressed by more easily giving commit/merge
access to these tools than to the main PG repo. And I think 2 could be
addressed by writing on the relevant wiki page where to go, and
probably putting a link to the wiki page on the actual website of the
tool.

But Perl is at the next level of unmaintained infrastructure. It is
actually clear how you can contribute to it, but still no new
community members actually want to contribute to it. Also, it's not
only unmaintained by us but it's also pretty much unmaintained by the
upstream community.

> But I still
> think it's fair to question whether the preference of many developers
> for Python over Perl will translate into sustained investment in
> improving the infrastructure. Again, I will be thrilled if it does,
> but that just doesn't seem to be the way that things go around here,
> and I bet the reasons go well beyond choice of programming language.

As you said, no one in our community wants to maintain our testsuite
full time. But our test suite consists partially of upstream
dependencies and partially of our own code. Right now pretty much
no-one improves the ustream code, and pretty much no-one improves our
own code. Using a more modern language gives up much more frequent
upstream improvements for free, and it will allow new community
members to contribute to our own test suite.

And I understand you are sceptical that people will contribute to our
own test suite, just because it's Python. But as a counterpoint:
people are currently already doing exactly that, just outside of the
core postgres repo[1][2][3]. I don't see why those people would
suddenly stop doing that if we include such a suite in the official
repo. Apparently many people hate writing tests in Perl so much that
they'd rather build Python test frameworks to test their extensions,
than to use/improve the Perl testing framework included in Postgres.

[1]: https://github.com/pgbouncer/pgbouncer/tree/master/test
[2]: https://github.com/jchampio/pg-pytest-suite
[3]: https://github.com/postgrespro/testgres


PS. I don't think it makes sense to host our tooling like the
commitfest app on our own git server instead of github/gitlab. That
only makes it harder for community members to contribute and also much
harder to set up CI. I understand the reasons why we use mailing lists
for the development of core postgres, but I don't think those apply
nearly as much to our tooling repos. And honestly also not to stuff
like the website.