Re: TAP output format in pg_regress

> On 4 Jul 2022, at 16:27, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:
>
> On 29.06.22 21:50, Daniel Gustafsson wrote:
>> Attached is a new version of this patch, which completes the TAP output format
>> option such that all codepaths emitting output are TAP compliant.  The verbose
>> option is fixed to to not output extraneous newlines which the previous PoC
>> did.  The output it made to conform to the original TAP spec since v13/14 TAP
>> parsers seem less common than those that can handle the original spec.  Support
>> for the new format additions should be quite simple to add should we want that.
>> Running pg_regress --verbose should give the current format output.
>> I did end up combining TAP and --verbose into a single patch, as the TAP format
>> sort of depends on the verbose flag as TAP has no verbose mode.  I can split it
>> into two separate should a reviewer prefer that.
>
> I'm not sure what to make of all these options.  I think providing a TAP output for pg_regress is a good idea.  But
thendo we still need the old output?  Is it worth maintaining two output formats that display exactly the same thing in
slightlydifferent ways? 

If we believe that TAP is good enough for human consumption and not just as
input to test runners then we don't.  Personally I think the traditional format
is more pleasant to read than raw TAP output when running tests.

> What is the purpose of the --verbose option?  When and how is one supposed to activate that?  The proposed default
formatnow hides the fact that some tests are started in parallel. 

The discussion on this was in 20220221164736.rq3ornzjdkmwk2wo@alap3.anarazel.de
where it was proposed that we could cut the boilerplate.  Thinking on it I
agree that the parallel run info should be included even without --verbose so
I'll add that back.  In general, running with --verbose should ideally only be
required for troubleshooting setup/teardown issues with testing.

>  I remember the last time I wanted to tweak the output of the parallel tests, people were very attached to the
particulartiming and spacing of the current output.  So I'm not sure people will like this. 
>
> The timing output is very popular.  Where is that in the TAP output?

As I mentioned in the mail upthread, TAP runners generally hide that info and
only show a running total.  That being said, I do agree with adding back so
I'll do that in a new version of the patch.

> More generally, what do you envision we do with this feature?  Who is it for, what are the tradeoffs, etc.?

In general, my thinking with this was that normal testruns started with make
check (or similar) by developers would use the traditional format (albeit less
verbose) and that the TAP output was for automated test runners in general and
the meson test runner in particular.  The TAP format is an opt-in with the
traditional format being the default.

The tradeoff is of course that maintaining two output formats is more work than
maintaining one, but it's not really something we change all that often so that
might not be too heavy a burden.

I personally didn't see us replacing the traditional format for "human
readable" runs, if that's where the discussion is heading then the patch can
look quite different.  Having test output format parity with supported back
branches seemed like a good idea to me at the time of writing at least.

--
Daniel Gustafsson        https://vmware.com/