Re: RFC: adding pytest as a supported test framework

On Tue, Jun 11, 2024 at 8:05 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2024-06-10 Mo 21:49, Andres Freund wrote:
>
> Hi,
>
> On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
>
> I'm not sure what part of the testing infrastructure you think is
> unmaintained. For example, the last release of Test::Simple was all the way
> back on April 25.
>
> IPC::Run is quite buggy and basically just maintained by Noah these days.
>
>
> Yes, that's true. I think the biggest pain point is possibly the recovery tests.
>
> Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we
couldprobably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It
wouldn'tavoid all uses of IPC::Run, though. 
>
> But my point was mainly that while a new framework might have value, I don't think we need to run out and immediately
rewriteseveral hundred TAP tests. Let's pick the major pain points and address those. 

FWIW, I felt a lot of pain trying to write recovery TAP tests with
IPC::Run's pumping functionality. It was especially painful (as
someone who knows even less Perl than the "street fighting Perl"
Thomas Munro has described having) before the additional test
infrastructure was added in BackgroudPsql.pm last year. As an example
of the "worst case", it took me two full work days to go from a repro
(with psql sessions on a primary and replica node) of the vacuum hang
issue being explored in [1] to a sort-of working TAP test which
demonstrated it - and that was with help from several other
committers. Granted, this is a complex case.

A small part of the issue is that, as Tristan has said elsewhere,
there aren't good developer tool integrations that I know about for
Perl. I use neovim's LSP support for C and Python (in other projects),
and there is a whole ecosystem of tools I can use for both C and
Python. I know not everyone likes or needs these, but I find that they
help me write and debug code faster.

I had offered to take a stab at writing some of the BackgroundPsql
test infrastructure in Python. I haven't started exploring that yet or
looking at what Jacob has done so far, but I am optimistic that this
is an area where it is worth seeing what is available to us outside of
IPC::Run.

- Melanie

[1] https://www.postgresql.org/message-id/CAAKRu_bXH2g_pchG7rN_4fs-_6_kVbbJ97gYRoN0Zdb9P04Wag%40mail.gmail.com