Recently we started migrating our application from websphere-liberty 16.0.0.2 to version 17.0.0.2 (in both cases using javaee7 profile). With the same SSL configurations in server.xml the application is not able to invoke remote rest service over https due to SSL handshake failure. Here is my server.xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="Default server">
<featureManager>
<feature>appSecurity-2.0</feature>
<feature>transportSecurity-1.0</feature>
<feature>jaxrs-2.0</feature>
<feature>json-1.0</feature>
<feature>javaMail-1.5</feature>
<feature>ssl-1.0</feature>
</featureManager>
<sslDefault sslRef="defaultSSLSettings" />
<ssl id="defaultSSLSettings" keyStoreRef="defaultKeyStore" trustStoreRef="defaultTrustStore" clientAuthentication="true" sslProtocol="TLSv1" />
<keyStore id="defaultKeyStore" location="/opt/ibm/wlp/output/defaultServer/resources/security/key.jks" password="**********" />
<keyStore id="defaultTrustStore" location="/opt/ibm/wlp/output/defaultServer/resources/security/trust.jks" password="***********" />
<basicRegistry id="basic" realm="BasicRealm">
<!-- <user name="yourUserName" password="" /> -->
</basicRegistry>
<variable name="defaultHostName" value="default-host.com" />
<!-- To allow access to this server from a remote client host="*" has been added to the following element -->
<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080" httpsPort="9443" />
<!-- Automatically expand WAR files and EAR files -->
<applicationManager autoExpand="true"/>
<logging logDirectory="/var/log/wlp" traceSpecification="*=INFO:SSL=all" traceFileName="trace.log" consoleLogLevel="info" copySystemStreams="true"/>
</server>
The only difference in server.xml for websphere-liberty v.16 is that there is no transportSecurity-1.0 feature enabled.
The rest service that the application invokes over https provides SSL certificate signed by "GeoTrust Global CA". If import the certificate into the truststore 'trust.jks' then everything works fine but I expect the certificate to be automatically accepted during handshake since it's not self-signed.
What changes were done in websphere-liberty 17.0.0.2 related to SSL? Are there any additional security configurations should be done in server.xml?
Servers versions:
WebSphere Application Server 17.0.0.2 (1.0.17.cl170220170523-1818) on IBM J9 VM, version pxa6480sr4fp10-20170727_01 (SR4 FP10) (en_US)
WebSphere Application Server 16.0.0.2 (1.0.13.cl160220160526-2258) on IBM J9 VM, version pxa6480sr3fp10-20160720_02 (SR3 FP10) (en_US)
Error stack trace:
server_1 | [INFO ] FFDC1015I: An FFDC Incident has been created: "java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
server_1 | java.security.cert.CertPathValidatorException: The certificate issued by CN=GeoTrust Global CA, O=GeoTrust Inc., C=US is not trusted; internal cause is:
server_1 | java.security.cert.CertPathValidatorException: Certificate chaining error com.ibm.ws.ssl.core.WSX509TrustManager checkServerTrusted" at ffdc_17.08.09_14.50.00.0.log
server_1 | [ERROR ] CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=*.api.ibm.com, O=International Business Machines, L=Armonk, ST=New York, C=US was sent from the target host. The signer might need to be added to local trust store /opt/ibm/wlp/output/defaultServer/resources/security/trust.jks, located in SSL configuration alias defaultSSLSettings. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
server_1 | java.security.cert.CertPathValidatorException: The certificate issued by CN=GeoTrust Global CA, O=GeoTrust Inc., C=US is not trusted; internal cause is:
server_1 | java.security.cert.CertPathValidatorException: Certificate chaining error
server_1 | [WARNING ] Interceptor for {https://dev.api.ibm.com/scx/test}WebClient has thrown exception, unwinding now
server_1 | Could not send Message.
server_1 | [ERROR ] 2017-08-09 14:50:00 ExceptionMapper:23 - javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1 | javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1 | at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:632)
server_1 | at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:608)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1105)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1042)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:895)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:863)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:426)
server_1 | at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1554)
server_1 | at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1549)
server_1 | at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.get(WebClient.java:1469)
server_1 | at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.get(InvocationBuilderImpl.java:80)
server_1 | at com.ibm.si.saas.sbs.service.SBSApiService.getCustomer(SBSApiService.java:84)
server_1 | at com.ibm.si.saas.sbs.resource.SbsEndpointApiResource.getConsoles(SbsEndpointApiResource.java:49)
server_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
server_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
server_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
server_1 | at java.lang.reflect.Method.invoke(Method.java:508)
server_1 | at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:632)
server_1 | at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:118)
server_1 | at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
server_1 | at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:252)
server_1 | at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)
server_1 | at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:423)
server_1 | at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
server_1 | at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:61)
server_1 | at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:99)
server_1 | at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
server_1 | at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:124)
server_1 | at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
server_1 | at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:134)
server_1 | at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:149)
server_1 | at com.ibm.websphere.jaxrs.server.IBMRestServlet.doGet(IBMRestServlet.java:115)
server_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
server_1 | at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:99)
server_1 | at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1290)
server_1 | at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:778)
server_1 | at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
server_1 | at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1161)
server_1 | at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4983)
server_1 | at com.ibm.ws.webcontainer31.osgi.webapp.WebApp31.handleRequest(WebApp31.java:528)
server_1 | at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:315)
server_1 | at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1025)
server_1 | at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:280)
server_1 | at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:967)
server_1 | at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:359)
server_1 | at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:318)
server_1 | at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:471)
server_1 | at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:405)
server_1 | at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:285)
server_1 | at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:256)
server_1 | at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:174)
server_1 | at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:83)
server_1 | at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
server_1 | at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
server_1 | at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
server_1 | at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
server_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
server_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
server_1 | at java.lang.Thread.run(Thread.java:785)
server_1 | Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
server_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:83)
server_1 | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
server_1 | at java.lang.reflect.Constructor.newInstance(Constructor.java:437)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1385)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1369)
server_1 | at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
server_1 | at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:653)
server_1 | at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
server_1 | at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
server_1 | at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:704)
server_1 | at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1041)
server_1 | ... 55 more
server_1 | Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1 | at com.ibm.jsse2.k.a(k.java:17)
server_1 | at com.ibm.jsse2.at.a(at.java:851)
server_1 | at com.ibm.jsse2.D.a(D.java:333)
server_1 | at com.ibm.jsse2.D.a(D.java:113)
server_1 | at com.ibm.jsse2.E.a(E.java:79)
server_1 | at com.ibm.jsse2.E.a(E.java:107)
server_1 | at com.ibm.jsse2.D.r(D.java:610)
server_1 | at com.ibm.jsse2.D.a(D.java:372)
server_1 | at com.ibm.jsse2.at.a(at.java:558)
server_1 | at com.ibm.jsse2.at.i(at.java:73)
server_1 | at com.ibm.jsse2.at.a(at.java:357)
server_1 | at com.ibm.jsse2.at.startHandshake(at.java:723)
server_1 | at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:215)
server_1 | at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:34)
server_1 | at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1561)
server_1 | at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489)
server_1 | at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:491)
server_1 | at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:80)
server_1 | at org.apache.cxf.transport.http.URLConnectionHTTP
server_1 | Conduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:370)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1586)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1615)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1559)
server_1 | at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1356)
server_1 | ... 61 more
server_1 | Caused by: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1 | at com.ibm.ws.ssl.core.WSX509TrustManager.checkServerTrusted(WSX509TrustManager.java:322)
server_1 | at com.ibm.jsse2.az.checkServerTrusted(az.java:78)
server_1 | at com.ibm.jsse2.E.a(E.java:5)
server_1 | ... 79 more
Here is output when trying to connect using openssl:
openssl s_client -CAfile /opt/ibm/java/jre/lib/security/cacerts -connect dev.api.ibm.com:443
Certificate chain
0 s:/C=US/ST=New York/L=Armonk/O=International Business Machines/CN=*.api.ibm.com
i:/C=US/O=GeoTrust Inc./CN=GeoTrust SSL CA - G3
1 s:/C=US/O=GeoTrust Inc./CN=GeoTrust SSL CA - G3
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2859 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 9509AB7194AB9DE0ABE219ECAE442C5D2C93145946FC2285CB9C4F5CCC81514F
Session-ID-ctx:
Master-Key: 335D047C41E5FE75096E53C9CBACC7C1CC8F6254872599EFE93C7AD5935638181AAF9240656BD44A858723C38108BB31
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1502310152
Timeout : 300 (sec)
Verify return code: 0 (ok)
The way you describe things it sound like when you were running Liberty 16.0.0.2 the JSSE SSL Context was being used in your scenario. If the cacerts file was being used for trust then it must have been. To be honest I thought jaxrs should have fallen back to the Liberty SSLContext if you do not provide a ssl reference in your code. I could be wrong about that and will have to investigate.
So at 17.0.0.2 and using transportSecurity-1.0 feature you are now using the a Liberty SSLContext. This is the correct behavior with transportSecurity-1.0, jaxrs will only fall back to the JSSE default SSLContext if there is no ssl configuration in the server.xml file. The transportSecurity-1.0 feature works like that to support the new outbound SSL features, outbound default, https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_config_ssl_outbound.html and outbound SSL Filter feature, https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_config_ssl_outbound_filter.html.
So if you use the transportSecurity-1.0 feature you will need to add the signer to you Liberty truststore, Liberty does not do any auto accepting of signers.
I will have to investigate what the behavior should be if the ssl-1.0 feature is used.