I have a cert file that is working fine under one Open Liberty instance (version 20.0.0.2 under jdk-11.0.5+10-openj9) to make outbound calls, using the standard HttpsURLConnection, to a particular endpoint that uses cert-auth. On a separate server of the same version/JDK, I am creating a new app using the MicroProfile REST Client APIs (using @Asynchronous, if that matters) but calls are failing with handshake failure. I am using the same cert and the same keystore config (and have enabled the "ssl-1.0" feature in both servers)...
<keyStore id="defaultKeyStore" location="key.jks" password="changeit" type="jks"/>
I tried debugging using "-Djavax.net.debug=all" under both servers and found that the cert with my target alias is found and added as trusted certificates in both cases upon server startup....
SunX509KeyManagerImpl.java:164|found key for : my-alias (...)
X509TrustManagerImpl.java:79|adding as trusted certificates (...)
SSLContextImpl.java:115|trigger seeding of SecureRandom
SSLContextImpl.java:119|done seeding of SecureRandom
But, for the mpRestClient app, when I make a call to the endpoint in question it seems to spontaneously switch trust stores to the default JDK cacerts trust store...
TrustStoreManager.java:112|trustStore is: C:\Program Files\AdoptOpenJDK\jdk-11.0.5+10-openj9\lib\security\cacerts
....
TrustStoreManager.java:311|Reload the trust store
TrustStoreManager.java:318|Reload trust certs
TrustStoreManager.java:323|Reloaded 88 trust certs
X509TrustManagerImpl.java:79|adding as trusted certificates (...)
SSLContextImpl.java:115|trigger seeding of SecureRandom
SSLContextImpl.java:119|done seeding of SecureRandom
After a bunch of negotiation happens, it eventually comes to...
CertificateMessage.java:290|No X.509 certificate for client authentication, use empty Certificate message instead
CertificateMessage.java:321|Produced client Certificate handshake message (
"Certificates": <empty list>
)
....
TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
"throwable" : {
javax.net.ssl.SSLException: readHandshakeRecord
....
}
)
Of course, there's no cert found for client auth because the whole truststore got reloaded, wiping out the originally loaded truststore. This spontaneous switching of trust stores does not occur under the other server. The relevant, successful, behavior on that server is...
SunX509KeyManagerImpl.java:401|matching alias: my-alias
ServerHelloDone.java:151|Consuming ServerHelloDone handshake message (
<empty>
)
CertificateMessage.java:321|Produced client Certificate handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 3769
SSLSocketOutputRecord.java:255|Raw write (...)
RSAClientKeyExchange.java:193|Produced RSA ClientKeyExchange handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 262
SSLSocketOutputRecord.java:255|Raw write (...)
CertificateVerify.java:743|Produced CertificateVerify handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 264
SSLSocketOutputRecord.java:255|Raw write (...)
ChangeCipherSpec.java:115|Produced ChangeCipherSpec message
SSLSocketOutputRecord.java:225|Raw write (...)
Finished.java:398|Produced client Finished handshake message (...)
....
ChangeCipherSpec.java:149|Consuming ChangeCipherSpec message
....
SSLSocketInputRecord.java:249|READ: TLSv1.2 handshake, length = 64
SSLCipher.java:1329|Padded plaintext after DECRYPTION (...)
Finished.java:535|Consuming server Finished handshake message (...)
SSLSocketOutputRecord.java:309|WRITE: TLS12 application_data, length = 339
SSLCipher.java:1483|Padded plaintext before ENCRYPTION (...)
SSLSocketOutputRecord.java:323|Raw write (...)
...
SSLSocketInputRecord.java:249|READ: TLSv1.2 application_data, length = 544
SSLCipher.java:1329|Padded plaintext after DECRYPTION (...)
[...and then I get my decrypted response...]
Another difference between the two server setups is that the successful one is a standalone download of the javaee8 version of Open Liberty while the unsuccessful one is executed via Maven...
mvn liberty:dev -Ddebug=false -DskipTests=true
I don't know why that should make any difference, but something clearly is. The configs are set up nearly identically. I've been searching far and wide and trying everything I could think of to tinker with, but now I'm just at a loss for what could be making the difference. Hopefully someone notices something. The key is to get the server running the mpRestClient app to stop spontaneously switching trust stores to the JDK cacerts default. But nothing I've tried prevents that so far.
Ultimately, I think this is a bug, so I'd recommend that you open an issue at https://github.com/OpenLiberty/open-liberty/issues (please be sure to reference this page). I think you can work around this issue by adding in the appSecurity-2.0 feature. It seems like JAX-RS requires both the SSL and AppSecurity features in order to use a different key store. That explains the ClassNotFoundException - the JaxRsSSLManager class is only added to the framework's classpath if the appSecurity-2.0 (or 3.0) feature is installed.
Using the AppSecurity feature should resolve the issue, but I consider that to be a workaround. If you open an issue with OpenLiberty, we can try to get this working without the AppSecurity feature.
Hope this helps, Andy