I have written a Netty-based REST service which is deployed directly (i.e. without a Web/Application server). For authn, authz and session management I am using Apache Shiro. Now I want to be able to load-balance this service without using sticky sessions on our load-balancer; and I am trying to set that up using in-process Hazelcast instances.
So far I have managed to have two instances of my service running on two different machines create a Hazelcast cluster with two members; but even so, if I authenticate on one machine, the other still doesn't recognize the session. So I think I have probably made a mistake in my shiro.ini and/or my hazelcast.xml configuration files.
I am essentially using the HazelCastCacheManager class from here https://github.com/stormpath/shiro-hazelcast-web-sample/blob/master/src/main/java/com/stormpath/samples/shiro/hazelcast/cache/HazelcastCacheManager.java
Here is my hazelcast.xml (our setup doesn't allow multicast, I have to specify the machine IPs directly):
<hazelcast>
<properties>
<property name="hazelcast.logging.type">slf4j</property>
</properties>
<map name="shiro-activeSessionsCache">
<async-backup-count>1</async-backup-count>
<time-to-live-seconds>600</time-to-live-seconds>
</map>
<network>
<join>
<multicast enabled="false"></multicast>
<tcp-ip enabled="true">
<member>x.x.x.x:8050</member> <!-- server A -->
<member>x.x.x.x:8050</member> <!-- server B -->
</tcp-ip>
</join>
<interfaces enabled="true">
<interface>x.x.x.*</interface>
</interfaces>
<port port-count="20" auto-increment="false">8050</port>
</network>
</hazelcast>
And a portion of my shiro.ini
sessionDAO = org.apache.shiro.session.mgt.eis.EnterpriseCacheSessionDAO
sessionManager = org.apache.shiro.session.mgt.DefaultSessionManager
sessionDAO.activeSessionsCacheName = shiro-activeSessionsCache
securityManager.sessionManager.sessionDAO = $sessionDAO
cacheManager = HazelcastCacheManager
securityManager.cacheManager = $cacheManager
sessionManager.globalSessionTimeout = 600000
sessionValidationScheduler = org.apache.shiro.session.mgt.ExecutorServiceSessionValidationScheduler
sessionValidationScheduler.interval = 600000
sessionManager.sessionValidationScheduler = $sessionValidationScheduler
securityManager.sessionManager=$sessionManager
What am I missing? How can I instruct Shiro to share the session among all Hazelcast instances? Or is my mistake in the HazelcastCacheManager class?
Here is the log Hazelcast produces when I start up my two services:
This is the first service starting up:
log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 24, 2015 9:03:42 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Interfaces is enabled, trying to pick one address matching to one of: [x.x.x.*]
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Prefer IPv4 stack is true.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Picked Address[x.x.x.x]:8050, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=8050], bind any local is true
[main] INFO com.hazelcast.spi.OperationService - [x.x.x.x]:8050 [dev] [3.5.3] Backpressure is disabled
[main] INFO com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor - [x.x.x.x]:8050 [dev] [3.5.3] Starting with 16 generic operation threads and 32 partition operation threads.
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Hazelcast 3.5.3 (20151011 - 64c663a) starting at Address[x.x.x.x]:8050
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
[main] INFO com.hazelcast.instance.Node - [x.x.x.x]:8050 [dev] [3.5.3] Creating TcpIpJoiner
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTING
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Connecting to /x.x.x.x:8050, timeout: 0, bind-any: true
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Could not connect to: /x.x.x.x:8050. Reason: SocketException[Connection refused to address /x.x.x.x:8050]
[cached1] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is added to the blacklist.
[main] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [x.x.x.x]:8050 [dev] [3.5.3]
Members [1] {
Member [x.x.x.x]:8050 this
}
And this is the continuation when the second service starts:
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTED
[main] INFO org.apache.shiro.config.IniSecurityManagerFactory - Realms have been explicitly set on the SecurityManager instance - auto-setting of realms will not occur.
Open your web browser and navigate to https://127.0.0.1:8113/
[hz._hzInstance_1_dev.IO.thread-Acceptor] INFO com.hazelcast.nio.tcp.SocketAcceptor - [x.x.x.x]:8050 [dev] [3.5.3] Accepting socket connection from /x.x.x.x:35316
[cached2] INFO com.hazelcast.nio.tcp.TcpIpConnectionManager - [x.x.x.x]:8050 [dev] [3.5.3] Established socket connection between /x.x.x.x:8050
[hz._hzInstance_1_dev.generic-operation.thread-10] INFO com.hazelcast.cluster.ClusterService - [x.x.x.x]:8050 [dev] [3.5.3]
Members [2] {
Member [x.x.x.x]:8050 this
Member [x.x.x.x]:8050
}
This is the second service starting up, with the first one already running:
log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 24, 2015 9:04:16 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Interfaces is enabled, trying to pick one address matching to one of: [x.x.x.*]
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Prefer IPv4 stack is true.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Picked Address[x.x.x.x]:8050, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=8050], bind any local is true
[main] INFO com.hazelcast.spi.OperationService - [x.x.x.x]:8050 [dev] [3.5.3] Backpressure is disabled
[main] INFO com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor - [x.x.x.x]:8050 [dev] [3.5.3] Starting with 8 generic operation threads and 16 partition operation threads.
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Hazelcast 3.5.3 (20151011 - 64c663a) starting at Address[x.x.x.x]:8050
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
[main] INFO com.hazelcast.instance.Node - [x.x.x.x]:8050 [dev] [3.5.3] Creating TcpIpJoiner
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTING
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Connecting to /x.x.x.x:8050, timeout: 0, bind-any: true
[cached1] INFO com.hazelcast.nio.tcp.TcpIpConnectionManager - [x.x.x.x]:8050 [dev] [3.5.3] Established socket connection between /x.x.x.x:35316
[hz._hzInstance_1_dev.generic-operation.thread-3] INFO com.hazelcast.cluster.ClusterService - [x.x.x.x]:8050 [dev] [3.5.3]
Members [2] {
Member [x.x.x.x]:8050
Member [x.x.x.x]:8050 this
}
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTED
[main] INFO org.apache.shiro.config.IniSecurityManagerFactory - Realms have been explicitly set on the SecurityManager instance - auto-setting of realms will not occur.
Open your web browser and navigate to https://127.0.0.1:8113/
I think my error was in my actual Java code, not in the configuration files:
In all my initial tests, running the service on a single server, I used this to get the current Shiro user:
Subject currentUser = SecurityUtils.getSubject();
However, the session key produced by this is the same for the same user on one server, but different for the same user on different servers; and the session key is used to identify a session in the cache. I was originally hoping to be able not having to pass a session key along with a REST call. but if I explicitly pass the session key and then use this to get the current user, then sessions can successfully be recognized in different servers:
Subject currentUser = null;
if (parameters.containsKey("session")) {
sId = parameters.get("session");
currentUser = new Subject.Builder().sessionId(sId).buildSubject();
} else {
currentUser = SecurityUtils.getSubject();
}
(I am using a Map to store all URL parameters before reaching this code snippet)
Follow-up question: is is possible to share Shiro sessions without cookies or explicitly passing the session key in other ways from the client to the server?