pcks11-provider doesn't work when FIPS mode is enabled

[GeorgePantelakis]

Describe the bug
When we are running pcks11-provider and FIPS mode is enabled, the provider is loaded by OpenSSL but none of the functions are FIPS compatible so none of them is called or used. Trying to load a pkcs11 URI makes OpenSSL fall back to the default provider and try to open it as a file. This fails and the "Failed to open OpenSSL store: error:8000000D:system library::Permission denied" error message is printed.

To Reproduce
We are using libssh API to reproduce this bug.
Steps to reproduce the behavior:

1. Create an RSA key-pair with OpenSSL
2. Create softhsm tokens from the key-pair
3. Export "PKCS11_PROVIDER_MODULE=/usr/lib64/pkcs11/libsofthsm2.so"
4. Run the test harness with the pksc11 URI from the tokens
5. See error in the output

Expected behavior
The pcks11 keys (both RSA and ECDSA) should be able to be loaded by the pcks11-provider in FIPS mode.

Operating environment (please complete the following information):

• OS: RedHat Enterprise Linux
• Version: 10

Token and application used (please complete the following information):

• Device: softhsm tokens
• PKCS11 Driver version: 0.5
• Application: libssh API
• Version: 0.11.1

Harness
libssh API C code converted to txt. Feel free to reproduce it however you see fit.
simple-libssh-client.txt

[simo5]

Isn't this basically a duplicate of #164 ?

[beldmit]

It is - but #164 was basically forgotten

[Jakuje]

Discussed this today with the following outcomes:

In short term (el10) we will need a configuration option that will be required to set by the user meaning "the inserted token is FIPS compliant". This will allow providing the OpenSSL with interfaces with "fips=yes" and it will allow applications to use them. We used to have something similar as part of the engine pkcs11 (but it was hardcoded):

https://src.fedoraproject.org/rpms/openssl-pkcs11/blob/rawhide/f/openssl-pkcs11-0.4.10-set-rsa-fips-method-flag.patch

In the long term, we will need two things:

• The openssl to support callbacks, that will allow us to invalidate caches Broken providers no_cache support openssl/openssl#26038
• The PKCS#11 3.2 specs that will allow signalization from the pkcs11 tokens to the provider whether the given operation is FIPS compliant
• (the pkcs11 drivers actually implementing the PKCS#11 3.2)