> On 3 Apr 2024, at 20:09, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 30.03.24 00:14, Daniel Gustafsson wrote:
>> One take-away for me is how important it is to ship recipes for regenerating
>> any testdata which is included in generated/compiled/binary format. Kind of
>> how we in our tree ship the config for test TLS certificates and keys which can
>> be manually inspected, and used to rebuild the testdata (although the risk for
>> injections in this particular case seems low). Bad things can still be
>> injected, but formats which allow manual review at least goes some way towards
>> lowering risk.
>
> I actually find the situation with the test TLS files quite unsatisfactory. While we have build rules, the output
filesare not reproducible, both because of some inherent randomness in the generation, and because different OpenSSL
versionsproduce different details.
This testdata is by nature not reproducible, and twisting arms to make it so
will only result in testing against synthetic data which risk hiding bugs IMO.
> So you just have to "trust" that what's there now makes sense.
Not entirely, you can review the input files which are used to generate the
test data and verify that those make sense..
> Of course, you can use openssl tools to unpack these files, but that is difficult and unreliable unless you know
exactlywhat you are looking for.
..and check like you mention above, but it's as you say not entirely trivial. It
would be nice to improve this but I'm not sure how. Document how to inspect
the data in src/test/ssl/README perhaps?
> It would be better if we created the required test files as part of the test run. (Why not? Too slow?)
The make sslfiles step requires OpenSSL 1.1.1, which is higher than what we
require to be installed to build postgres. The higher-than-base requirement is
due to it building test data only used when running 1.1.1 or higher, so
technically it could be made to work if anyone is interesting in investing time
in 1.0.2.
Time is one aspect, on my crusty old laptop it takes ~2 seconds to regenerate
the files. That in itself isn't that much, but we've rejected test-time
additions far less than that. We could however make CI and the Buildfarm run
the regeneration and leave it up to each developer to decide locally? Or
remove them and regenerate on the first SSL test run and then use the cached
ones after that?
On top of time I have a feeling the regeneration won't run on Windows. When
it's converted to use Meson then that might be fixed though.
--
Daniel Gustafsson