Hi hackers,
While working on testing post-quantum cryptography integration in my system, I discovered that PostgreSQL has an issue with channel binding when using ML-DSA cryptographic algorithms.
The problem is caused by a difference between the currently used algorithms and post-quantum ones. For example, commonly used algorithms like RSA have a defined digest algorithm, but ML-DSA does not.
PostgreSQL's channel binding implementation expects all signature algorithms to have a traditional digest mapping, but post-quantum algorithms such as ML-DSA use their hash function internally as part of the signature process.
As a result, the connection fails with the following error:
could not find digest for NID UNDEF
The issue can be worked around by disabling channel binding.
Although the RFC is not entirely clear on how to handle this situation, in my patch I propose using SHA-256 as the default digest in such cases.