TPM 2.0 based TLS handshake fails against RSA-4k Server keys (out-of-range)

I have an Embedded Client w/ an TPM 2.0 that needs to connect to a remote Server using mTLS and a TPM-based private key. The client is also fitted with a file-based (PEM) client cert from a CA accepted by the Server.

To integrate with the TPM, the intention is to use the OpenSSL TPM Provider (tpm2-openssl)

I tried to follow the example from the tpm2-openssl docs: https://github.com/tpm2-software/tpm2-openssl/blob/master/docs/certificates.md

I'm currently running this on a Xubuntu 23.10 VirtualBox VM with an emulated TPM 2.0 (version info at the bottom).

What does work

It works when I start a Server with RSA-2k keys and try to connect to it using OpenSSL's s_server and s_client:

openssl s_server -accept 4443 -www -key server-2k.key -cert server-2k.crt

openssl s_client -provider tpm2 -provider default -propquery '?provider=tpm2' \
-connect localhost:4443 -CAfile ca.crt -cert client.crt -key client-tpm.priv

What does not work

When I switch to RSA-4k Server keys (like the real Server uses), i'll catch the following out-of-range TPM error:

openssl s_server -accept 4443 -www -key server-4k.key -cert server-4k.crt

WARNING:esys:src/tss2-esys/api/Esys_LoadExternal.c:314:Esys_LoadExternal_Finish() Received TPM Error 
ERROR:esys:src/tss2-esys/api/Esys_LoadExternal.c:108:Esys_LoadExternal() Esys Finish ErrorCode (0x000002c4) 
40F71A26BC730000:error:4000000C:tpm2::cannot load key::-1:708 tpm:parameter(2):value is out of range or is not correct for the context
40F71A26BC730000:error:0A080006:SSL routines:tls_process_key_exchange:EVP lib:../ssl/statem/statem_clnt.c:2284:

I noticed that I observe the very same error message with the same code lines mentioned in the Load_External() function when I try to let a TPM-based private key sign CSR w/ a RSA-4k client key:

openssl x509 -req -provider tpm2 -provider default -in test-4k.csr \
-CA ca.crt -CAkey ca.priv -out test-4k.crt

What I wonder about

My assumption is that the issue arises from the missing RSA-4k support in TPM 2.0. I guess the TPM Provider tries to load the Server PubKey into the TPM, and the TPM will not accept RSA-4k keys.

However, we have a running stack using the older OpenSSL Engine (https://github.com/tpm2-software/tpm2-tss-engine) and never recognized an issue with RSA-4k keys so far.

I wonder...

is my line of thoughts correct about why it fails for RSA-4k Server keys?
why does that work with the older OpenSSL Engine API?

Version info

$ openssl version
OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)

$ sudo tpm2_getcap properties-fixed
TPM2_PT_FAMILY_INDICATOR:
  raw: 0x322E3000
  value: "2.0"
...
TPM2_PT_REVISION:
  raw: 0xA4
  value: 1.64
...
TPM2_PT_MANUFACTURER:
  raw: 0x49424D00
  value: "IBM"

Solution

4k keys are not supported by tpm.

This relates not only to the cert itself, but the whole chain. Once you specify tpm2 as a provider it will be used to veryfy ALL certs in the chain instead of just doing the handshake and signing. So a case where your cert is 2k and the CA is 4k will fail too.