This is part of the larger project of allowing all test suites to pass
in OpenSSL FIPS mode. We had previously integrated several patches that
avoid or isolate use of MD5 in various forms in the tests. Now to
another issue.
OpenSSL in FIPS mode rejects several encrypted private keys used in the
test suites ssl and ssl_passphrase_callback. The reason for this is
explained in [0]:
> Technically you shouldn't use keys created outside FIPS mode in FIPS
> mode.
>
> In FIPS mode the "traditional" format is not supported because it used
> MD5 for key derivation. The more standard PKCS#8 mode using SHA1 for
> key derivation is use instead. You can convert keys using the pkcs8
> command outside FIPS mode but again technically you aren't supposed
> to...
[0]:
https://groups.google.com/g/mailing.openssl.users/c/Sd5E8VY5O2s/m/QYGezoQeo84J
The affected files are
src/test/modules/ssl_passphrase_callback/server.key
src/test/ssl/ssl/client-encrypted-pem.key
src/test/ssl/ssl/server-password.key
A fix is to convert them from their existing PKCS#1 format to the PKCS#8
format, like this:
openssl pkcs8 -topk8 -in
src/test/modules/ssl_passphrase_callback/server.key -passin pass:FooBaR1
-out src/test/modules/ssl_passphrase_callback/server.key.new -passout
pass:FooBaR1
mv src/test/modules/ssl_passphrase_callback/server.key.new
src/test/modules/ssl_passphrase_callback/server.key
etc.
(Fun fact: The above command also doesn't work if your OpenSSL
installation is in FIPS mode because it will refuse to read the old file.)
We should also update the generation rules to generate the newer format,
like this:
- $(OPENSSL) rsa -aes256 -in server.ckey -out server.key -passout
pass:$(PASS)
+ $(OPENSSL) pkey -aes256 -in server.ckey -out server.key -passout
pass:$(PASS)
I have attached two patches, one to update the generation rules, and one
where I have converted the existing test files. (I didn't generate them
from scratch, so for example
src/test/modules/ssl_passphrase_callback/server.crt that corresponds to
one of the keys does not need to be updated.)
To check that these new files are backward compatible, I have
successfully tested them on CentOS 7 with the included version 1.0.2k.
It's also interesting that if you generate all private keys from scratch
using the existing rules on a new OpenSSL version (3+), they will be
generated in PKCS#8 format by default. In those OpenSSL versions, the
openssl-rsa command has a -traditional option to get the old format, but
of course old OpenSSL versions don't have that. As OpenSSL 3 gets more
widespread, we might need to rethink these rules anyway to make sure we
get consistent behavior.