I have a project built with Spring and Cassadra db. Actually, setting the client enrcyption in cassandra.yml to false all works.
Working setup (using spring XML beans and client encyption set to false)
<cassandra:cluster contact-points="${cassandra.contactpoints}"
port="${cassandra.port}"
username="${cassandra.username}"
password="${cassandra.password}" />
<cassandra:session id="cassandraSession"
keyspace-name="${cassandra.keyspace}" />
<cassandra:mapping />
<cassandra:converter />
<cassandra:template id="cassandraTemplate" />
Now I want to enable client-to-node encryption (node-to-node is already set and working). I found some tutorial online that explain how to create the needed keystore but i was unable to find out how to set up a Spring project to deal with it.
I have enabled client-to-node encryption in cassandra.yml and tried to set ssl-enabled="true" in cluster bean but I'm unable to connect to cassandra from Spring. I know there is an ssl-options-ref attribute in cassandra cluster bean but i can't find any tutorial on how to use this.
Also tried to follow this: How to set System Properties on run time Spring 3 MVC to set up system properties at run time to load correct truststore file, but adding that code makes no difference. I get always this error:
2016-08-19 13:46:26 INFO NettyUtil:83 - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2016-08-19 13:46:26 WARN DefaultPromise:151 - An exception was thrown by com.datastax.driver.core.Connection$10.operationComplete()
java.util.concurrent.RejectedExecutionException: Task com.datastax.driver.core.Connection$10$1@5c21d76e rejected from java.util.concurrent.ThreadPoolExecutor@78eaecc1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:556)
at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:573)
at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:547)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
at io.netty.channel.PendingWriteQueue.safeFail(PendingWriteQueue.java:286)
at io.netty.channel.PendingWriteQueue.removeAndFailAll(PendingWriteQueue.java:132)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1231)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1205)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1060)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:900)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at java.lang.Thread.run(Thread.java:745)
2016-08-19 13:46:26 INFO DefaultListableBeanFactory:444 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@51cc1210: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,cassandraCluster,cassandraSession,cassandraMapping,cassandraConverter,cassandraTemplate,smsBehavior,mailSender,preConfiguredMessage,recordingBehavior,user,trustStore]; root of factory hierarchy
2016-08-19 13:46:28 ERROR ContextLoader:331 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession': Invocation of init method failed; nested exception is com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: ******** (com.datastax.driver.core.exceptions.TransportException: [*******] Channel has been closed))
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:610)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:535)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1461)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1445)
at org.apache.catalina.manager.ManagerServlet.deploy(ManagerServlet.java:860)
at org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:357)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: ************* (com.datastax.driver.core.exceptions.TransportException: [*********] Channel has been closed))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1424)
at com.datastax.driver.core.Cluster.init(Cluster.java:163)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:334)
at com.datastax.driver.core.Cluster.connect(Cluster.java:284)
at org.springframework.cassandra.config.CassandraCqlSessionFactoryBean.connect(CassandraCqlSessionFactoryBean.java:100)
at org.springframework.cassandra.config.CassandraCqlSessionFactoryBean.afterPropertiesSet(CassandraCqlSessionFactoryBean.java:94)
at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.afterPropertiesSet(CassandraSessionFactoryBean.java:60)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
... 52 more
Supposing to have a SslOptionFactoryBean that extends AbstractFactoryBean<SSLOptions>
a sample configuration class to use it and initialize the cluster could be:
@Configuration
@PropertySource(value = {"file:./db.properties"})
public class CassandraConfig extends SpringHttpSessionConfiguration {
@Autowired
private SSLOptions sslOption;
@Bean
@Lazy(false)
public SslOptionFactoryBean sslOptions() throws URISyntaxException, IOException {
Resource trustStore = new FileSystemResource(env.getProperty("db.truststorefilename", ""));
Resource keyStore = new FileSystemResource(env.getProperty("db.keystorefilename", ""));
String trustStorePassword = env.getProperty("db.truststorepassword", "");
String keyStorePassword = env.getProperty("db.keystorepassword", "");
SslOptionFactoryBean option = new SslOptionFactoryBean();
option.setTrustStore(trustStore);
option.setTrustStorePassword(trustStorePassword);
option.setKeyStore(keyStore);
option.setKeyStorePassword(keyStorePassword);
return option;
}
@Bean
public Cluster cluster() throws Exception {
// load node address and port
/* ... */
return Cluster.builder()
.addContactPoint(node)
.withPort(port)
.withSSL(sslOption)
.build();
}
}
Set either the trust-store using System-properties outside the JVM (-Djavax.net.ssl.trustStore=…) or add a bean dependency on the System-Properties factory bean to make sure the properties are applied before the Cassandra client is initialized. Using ssl-options-ref requires more effort.
The System-Property based SSL configuration needs to be applied before the Datastax client is initialized. In fact, the properties should be applied as early as possible make sure no other class initializes the default SSL context because the default SSL context is cached. Applying javax.net.ssl.trustStore after any component initialized the default SSL context will not apply your settings.
You could use ssl-options-ref to provide dedicated SSL options with a configured SSL context, but this requires additional code. com.datastax.driver.core.SSLOptions cannot be just configured with property values. Please also note that upgrading the Cassandra driver to 3.0 requires a different SSL context initialization because the driver APIs changed.
A sample SslOptionsFactoryBean could look like:
public class SslOptionsFactoryBean extends AbstractFactoryBean<SSLOptions> {
private Resource keyStore;
private String keyStorePassword;
private Resource trustStore;
private String trustStorePassword;
@Override
public Class<?> getObjectType() {
return SSLOptions.class;
}
@Override
protected SSLOptions createInstance() throws Exception {
KeyManager[] keyManagers = getKeyStore() != null
? createKeyManagerFactory(getKeyStore(), getKeyStorePassword()).getKeyManagers() : null;
TrustManager[] trustManagers = getTrustStore() != null
? createTrustManagerFactory(getTrustStore(), getTrustStorePassword()).getTrustManagers() : null;
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);
return new SSLOptions(sslContext, SSLOptions.DEFAULT_SSL_CIPHER_SUITES);
}
private static KeyManagerFactory createKeyManagerFactory(Resource keystoreFile, String storePassword)
throws GeneralSecurityException, IOException {
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
try (InputStream inputStream = keystoreFile.getInputStream()) {
keyStore.load(inputStream, StringUtils.hasText(storePassword) ? storePassword.toCharArray() : null);
}
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keyStore, StringUtils.hasText(storePassword) ? storePassword.toCharArray() : new char[0]);
return keyManagerFactory;
}
private static TrustManagerFactory createTrustManagerFactory(Resource trustFile, String storePassword)
throws GeneralSecurityException, IOException {
KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
try (InputStream inputStream = trustFile.getInputStream()) {
trustStore.load(inputStream, StringUtils.hasText(storePassword) ? storePassword.toCharArray() : null);
}
TrustManagerFactory trustManagerFactory = TrustManagerFactory
.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(trustStore);
return trustManagerFactory;
}
public Resource getKeyStore() {
return keyStore;
}
public void setKeyStore(Resource keyStore) {
this.keyStore = keyStore;
}
public String getKeyStorePassword() {
return keyStorePassword;
}
public void setKeyStorePassword(String keyStorePassword) {
this.keyStorePassword = keyStorePassword;
}
public Resource getTrustStore() {
return trustStore;
}
public void setTrustStore(Resource trustStore) {
this.trustStore = trustStore;
}
public String getTrustStorePassword() {
return trustStorePassword;
}
public void setTrustStorePassword(String trustStorePassword) {
this.trustStorePassword = trustStorePassword;
}
}
The XML config part would look like:
<bean id="sslOptions" class="x.y.SslOptionsFactoryBean" lazy-init="false">
<property name="trustStore" value="file:truststore.jks"/>
</bean>
<cassandra:cluster contact-points="localhost"
port="9042"
username="user"
password="pass"
ssl-enabled="true"
ssl-options-ref="sslOptions"
/>
N.b.: The SslOptionsFactoryBean contains trust managers and key managers for a full SSLContext initalization.
The stack trace is showing that SSL is configured with the Datastax driver. It also shows that the handshake fails and the failure notification event fails with:
Task com.datastax.driver.core.Connection$10$1@5c21d76e rejected from java.util.concurrent.ThreadPoolExecutor@78eaecc1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
That message says the Netty EventLoopGroup is shut down during SSL handshake (failure) completion. The Spring container should run a bit longer to see the handshake failure message.