RE: On Judging the Value of Tests

>>>> 1.      How do you judge when a test suite is acceptable to be added to
>>>> Postgres OSS source code? (How do you judge the value of a test 
>>>> suite?)
>>>
>>> Make your argument for it, and see if others agree. There's no formal 
>>> process.
>
> Yeah.  Also realize that we've been accreting test cases for ~20 years,
> and there's never been any master plan about what to test.  So there's a 
> lot of individual test cases of varying quality/value, and you shouldn't
> necessarily take any one existing test as gospel. I don't hold that out 
> as a great example of software engineering, but it's the truth.
>>>
>>> Inputs into the decision making process include:
>>> * How much coverage of previously untested functionality it adds
>>> * How much code coverage it adds
>>> * How long the test takes to run, especially considering the slow 
>>> buildfarm boxes and development turnaround time
>>> * Whether the test fits into one of the existing suites we run 
>>> routinely, or requires separate steps
>>> * How much work will be required to maintain the test
>>
>> Even with those factors in mind there is a case-by-case variability.
>> Sometimes it is not worth initializing a new instance with initdb for
>> a small test. Parallelization of tests help on large machines by on 
>> buildfarm members the size of a RPI things get time-consuming.
>> Priority is usually given into things that do not consume much time, 
>> so as any developer can run the full set of tests quickly on a laptop.
>> I have for example a small alias to run the full set of basic tests 
>> (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within 
>> two minutes if I recall correctly, TAP tests included. So runtime is 
>> really an important matter.
>
> Also, portability/repeatability.  If it doesn't produce the same answers, 
> with very high probability, across all the platforms we support, it isn't 
> going to last long.

Thank you everyone for your valuable input.
I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not
coveredby tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still
learningthe PG internals and hacking.
 

>>>> 3.      Is there a standard way of writing tests on the source code that
>>>> we should follow, like when should test be written in TAP/SQL/C 
>>>> formats and how long should it be?
>>>
>>> In general, prefer pg_regress if you can use it. Isolation tests for 
>>> concurrency issues.  TAP tests if you can't write it with pg_regress 
>>> or isolation tester. Test modules only if you really must.
>
> Right.  This ties in closely to the incremental runtime needed for one 
> additional test case.  A new query or two in an existing SQL regression 
> script adds little overhead.  A new TAP test adds quite a bit.

I understand that tests should not take that long when running.
For example, how about when proposing tests that just test
parameter settings of a function? 
Let's say, for example, a libpq main function where in different test cases 
the params are null, empty, invalid, or valid.
Are these test types common or recommended?

>>>> 4.      In the src/test/examples directory (which are all libpq tests),
>>>> why is the “examples” directory not included when building postgres?
>>>
>>> Dunno, sorry.
>>
>> Those are mainly present as example programs.
>
> I've always taken those to be more documentation, or sample code, than
> test cases.  I'm not sure why they're under src/test/ at all.

Since these example codes are only documentations, so should these be removed in the source code instead? Or should
theseexamples code be revised and be moved to a test directory in libpq?
 

Thanks again!

Regards,
Kirk Jamison