Search code examples
spring-boothazelcastminikube

Hazelcast instance trying to access Kubernetes pods directly instead of load balancer's external IP

I have a Hazelcast cluster deployed in Minikube with 2 members. I have set up a load balancer service to access these members. My Spring Boot application, which is deployed outside of Minikube, uses the following configuration to initialize the Hazelcast instance:

public Config getHazelcastConfig() {
    Config config = new Config("default");

    config.getNetworkConfig().setPort(5701);
    config.getNetworkConfig().setPortCount(1);
    config.getNetworkConfig().setPortAutoIncrement(false);

    config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
    config.getNetworkConfig()
            .getJoin()
            .getTcpIpConfig()
            .addMember("external-ip:port")
            .setEnabled(true);
    return config;
}
@Bean
public HazelcastInstance hazelcastInstance() {
    Config config = getHazelcastConfig();
    return Hazelcast.newHazelcastInstance(config);
}

When I start the Spring Boot application for the first time after rebuilding it, I can successfully add the Minikube members to the cluster, resulting in a total of 3 members as expected. However, when I restart the application, it tries to access the Kubernetes pods directly using their private IP addresses. Here are the relevant logs:

2023-07-02 20:39:16.831  INFO 32243 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection     : [10.31.0.28]:5701 [dev] [4.2.1] Initialized new cluster connection between /192.168.105.1:50190 and /10.108.94.145:5701
2023-07-02 20:39:16.898  INFO 32243 --- [cached.thread-3] c.h.i.server.tcp.TcpServerConnector      : [10.31.0.28]:5701 [dev] [4.2.1] Connecting to /10.244.0.2:5701, timeout: 10000, bind-any: true
2023-07-02 20:39:26.902  INFO 32243 --- [cached.thread-3] c.h.i.server.tcp.TcpServerConnector      : [10.31.0.28]:5701 [dev] [4.2.1] Could not connect to: /10.244.0.2:5701. Reason: IOException[null to address /10.244.0.2:5701]
2023-07-02 20:39:26.903  INFO 32243 --- [cached.thread-3] c.h.internal.cluster.impl.TcpIpJoiner    : [10.31.0.28]:5701 [dev] [4.2.1] [10.244.0.2]:5701 is added to the blacklist.
2023-07-02 20:39:36.939  INFO 32243 --- [cached.thread-3] c.h.internal.cluster.impl.TcpIpJoiner    : [10.31.0.28]:5701 [dev] [4.2.1] [10.244.0.2]:5701 is added to the blacklist.

The IP 10.244.0.2 corresponds to one of the Kubernetes pods (members). However, my application should only access the members through the load balancer's external IP. I'm not sure why it's trying to access the pods directly.

How can I enforce my Hazelcast instance to always access the members via the load balancer's external IP? Any guidance or suggestions would be greatly appreciated.

Note: I am not using the ClientConfig because Im trying to add the members deployed in Kubernetes as new members along with the one member which starts with my spring boot application. I have setup my Minikube referring this Thank you.

Solution

  • Ongoing discussion on Hazelcast Community Slack

    Cluster members maintain open sockets to each other, a single LoadBalancer can 't make two members inside Kubernetes accessible to one member outside.