OpenSSL Communities

Behaviour of EVP_PKEY_get1_RSA in fips mode in opnessl 3.5.6 version

Ankit GuptaAnkit Gupta Mon 22 Jun 2026 8:29AMPublicSeen by 58

Hi,

I need some help regarding OpenSSL 3.5.6 in FIPS mode.

I have an EVP_PKEY containing an RSA key. When I call:

RSA* rsa = EVP_PKEY_get1_RSA(pkey);

the function returns NULL in FIPS mode.

My questions are:

  1. Is EVP_PKEY_get1_RSA() expected to return NULL in FIPS mode with provider-based keys?

  2. What troubleshooting steps should I perform to determine the root cause?

    • How can I verify whether the key is coming from the FIPS provider?

    • Are there specific error codes or provider checks I should perform?

  3. My code currently uses legacy RSA APIs such as:

    RSA_private_encrypt()
RSA_private_decrypt()

  4. What is the recommended replacement for these APIs when running in OpenSSL 3.x and FIPS mode?

    • Should I use EVP_PKEY_sign(), EVP_PKEY_decrypt(), EVP_PKEY_encrypt(), or another EVP-based API?

    • If the operation is actually being used for raw RSA signing rather than encryption, what is the correct migration path?

Additional information:

  • OpenSSL version: 3.5.6

  • FIPS provider enabled

  • The same code works when FIPS mode is disabled.

Any guidance would be appreciated.

Thanks.

Anton Arapov

Anton ArapovMon 27 Jul 2026 3:24PM

Hi Ankit,
Thanks for posting this — and sorry it sat here a while without a reply.

Honestly, questions like this are tough to settle on a public forum. FIPS-mode behaviour changing across versions usually comes down to the specifics of your build and module configuration, and getting to the bottom of it takes real back-and-forth with someone who knows the internals. That kind of close, private, hands-on work is what a commercial support arrangement with OpenSSL is there for — you work directly with the people who maintain the code, through to an actual fix rather than just an answer.

The community's always here for questions too, so please keep asking. I'll close this thread now.

Thanks again.