Enabling hybrid key exchange in FIPS mode for old FIPS providers

Cross-post from Distributions; Thread by @Dmitry Belyavsky:
OpenSSL 3.5 has enabled hybrid PQ key exchange with the new FIPS provider.
It's a great result but distributions (and I presume, big corporations) rely on previously certified versions of FIPS provider (e.g. 3.1 for upstream, 3.0.7 for RHEL).
According to a set of NIST documents (800-56Ar3) we can treat MLKEM as auxiliary data for EC key exchange.
Different orderings of the component data fields of FixedInfo may be used, and one or more of
the data fields may be combined (or omitted under certain circumstances). See Section 5 in SP
800-56C, and Sections 5, 7.4, 7.5 and 7.6 in SP 800-108 for details
We (Red Hat) has provided a proof-of-concept patch that works for our purpose and enables Hybrid kex in FIPS mode. It look a bit hackerish but resolves several obstacles:
pass propq down when generating MLKEM keys, so SHAKE would be fetched from default provider. Otherwise FIPS 3.0 provider doesn't have squeeze method before 3.2 (not certified upstream)
Expand EC pubkey when old (< 3.0.8) FIPS provider returns compressed one. We in RHEL have 3.0.7 so we came across this problem. I think it's acceptable upstream
I would like to see if this approach is of interest to the communities I represent (Distributions) or am involved in (Big corps)

Chris Brych Fri 20 Jun 2025 1:09PM
Oracle supports Red Hat's idea that enables hybrid KEX in FIPS mode.

Dimitri John Ledkov Fri 25 Jul 2025 3:28PM
Chainguard will support this interop too, and will also push for NIST resolution to approve this. We are looking to enable and test this interop across Go, Java, C implementations based on OpenSSL, Sun, Bouncycastle implementations within chainguard's implementations, as well as rhel and oracle products.
@ANUBHAV however hackish the patch, are you willing or interested to share it with me? the version numbers do not apply to stuff that Chaingaurd ships, and I would want to upstream something that allows to build extra providers which might make things work.

Dimitri John Ledkov Fri 25 Jul 2025 3:29PM
Note Keyfactor has developed Bouncycastle providers for Hybrid KEM that can be mixed/matches in a way to enable Hybrid KEM in such certification contexts too.
Jeff Johnson 路 Wed 18 Jun 2025 1:22PM
@Dmitry Belyavsky we have enabled hybrid kex in FIPS mode internally. This is a way to keep FIPS capable products until FIPS PQ algs in a module are approved. However, it should be noted that NIAP and Common Criteria do not like nor accept hybrid kex. Typically, those requirements were subsets of FIPS so FIPS "mode" was required prior to Common Criteria restrictions being enabled (a CC mode). Their preference for pure PQ algs has changed the way to approach that. Basically, you cannot use hybrid kex to meet NIAP/CC requirements (and hence not FIPS hybrid) and there are no modules yet that have pure PQ algs. So...... what do they do. Those orgs have conceded that in the meantime they will accept ACVP alg certs listed on NIST/CMVP websites as an interim solution. Lots of fun during this transtion! 馃檪