Re: RFC: adding pytest as a supported test framework

On Thu, Jun 13, 2024 at 8:12 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> But also: do you have opinions on what to fill in as steps 2
> (something we have no ability to test today) and 3 (something we do
> test today, but hate)?
>
> My vote for step 2 is "client and server separation", perhaps by
> testing libpq fallback against a server that claims support for
> different build-time options. I don't want to have a vote in step 3,
> because part of that step is proving that this framework can provide
> value for a part of the project I don't really know much about.

I mean, both Perl and Python are Turing-complete. Anything you can do
in one, you can do in the other, especially when you consider that
we're not likely to accept too many dependencies on external Perl or
Python modules. That's why I see this as nothing more or less than
exercise in letting people use the programming language they prefer.
We've talked about a libpq FFI interface, but it hasn't been done; now
we're talking about maybe having a Python one. Perhaps we'll end up
with both. Then you can imagine porting tests from one language to the
other and the only difference is whether you'd rather have line noise
before all of your variable names or semantically significant
whitespace.

I just don't believe in the idea that we're going to write one
category of tests in one language and another category in another
language. As soon as we open the door to Python tests, people are
going to start writing the TAP tests that they would have written in
Perl in Python instead. And if the test utilities that we have for
Perl are not there for Python, then they'll either open code things
for which they would have used a module, or they'll write a
stripped-down version of the module that will then get built-up patch
by patch until, 50 or 100 or 200 hours of committer-review later, it
resembles the existing Perl module. And, if the committer pushes back
and says, hey, why not write this test in Perl which already has all
of this test infrastructure in place already, then the submitter will
wander off muttering about how PostgreSQL committers are hidebound
backward individuals who just try to ruin everybody's day. So as I see
it, the only reasonable plan here if we want to introduce testing in
Python (or C#, or Ruby, or Go, or JavaScript, or Lua, or LOLCODE) is
to try to achieve a reasonable degree of parity between that language
and Perl. Because then we can at least review the new infrastructure
all at once, instead of incrementally spread across many patches
written, reviewed, and committed by many different people.

Now, I completely understand if you're not excited about getting
sucked down that rabbit-hole, and maybe some other committer is going
to see this differently than I do, and that's fine. But my view is
that if you're not interested in doing all the work to let people do
more or less the kind of stuff that they currently do in Perl in
Python instead, then your alternative is to take the tests that you
want to add and rewrite them in Perl. And I am fairly cetain that if
you choose that option, it will save me, and a bunch of other
committers, a whole lot of work, at least in the short term. If we add
support for Python, we are going to end up having to do a lot of
things twice for the next let's say ten to twenty years until somebody
rewrites the remaining Perl tests in Python or whatever language is
hot and cool by then. My opinion is that we need to be open to
enduring that pain because we can't indefinitely hold our breath and
insist on using obsolete tools for everything, but that doesn't mean
that I don't think it will be painful.

Consider the meson build system project. To get that committed, Andres
had to make it do pretty much everything MSVC could do and mostly
everything that configure could do, and the places where he didn't
make it do everything configure could do remain sore spots that I, at
least, am not happy about. And in that case, he also managed to get
MSVC removed entirely, so that we do not have a larger number of build
systems in total than we had before. Furthermore, the amount of code
in buildsystem files (makefiles, meson.build) in a typical patch
needing review is usually none or very little, whereas the amount of
test code in a patch is sometimes quite large. I've come around to the
belief that the meson work was worthwhile -- running tests is so much
faster and nicer now -- but it was a ton of work to get done and
impacted everyone in the development community, and I think the blast
radius for this change is likely to be larger for the reasons
suggested earlier in this paragraph.

--
Robert Haas
EDB: http://www.enterprisedb.com