Hazelcast not able to discover other nodes in Linux

We have the following Hazelcast.xml file

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.hazelcast.com/schema/config http://www.hazelcast.com/schema/config/hazelcast-config-3.11.xsd">
   <management-center enabled="false">http://localhost:8080/hazelcast-mancenter</management-center>
   <instance-name>DistributedCache</instance-name>
   <properties>
      <property name="hazelcast.logging.type">log4j2</property>
   </properties>
   <network>
      <interfaces enabled="true">
         <interface>10.5.167.*</interface>
      </interfaces>
   </network>
</hazelcast>

We have two different servers .10 and .11. When the application in both servers is started, a cluster is formed on port 5701. Here is the log output in both servers.

Server .10 log

Local member in cluster: Member [10.5.167.10]:5701 Non local members in cluster: [] Cluster size: 1

Server .11 log

Local member in cluster: Member [10.5.167.11]:5701 Non local members in cluster: [] Cluster size: 1

As can be seen Hazelcast is unable to discover the nodes in other servers (Non-local members is empty). There is no firewall between the two and there is also full connectivity on port 5701. We are able to telnet between both servers. Additionally in Windows we are able to observe that they are able to find each other, but not in Linux. That brings me to the question is there something we are missing here?

Additionally there is no mention of any differences when using tcp/ip Hazelcast discovery on Linux compare to windows in the documentation. Hazelcast version is 3.12.6

Solution

By default multicast discovery is used in Hazelcast. Try to switch to tcp-ip discovery instead.

   <network>
        <join>
            <multicast enabled="false"/>
            <tcp-ip enabled="true">
                <member-list>
                    <member>10.5.167.10</member>
                    <member>10.5.167.11</member>
                </member-list>
            </tcp-ip>
        </join>
   </network>

And your log message doesn't seem to be a Hazelcast original one. Usually you see ones such as:

Oct 27, 2022 8:40:53 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.5.167.10]:5701 [dev] [3.12.6] 

Members {size:2, ver:2} [
    Member [10.5.167.10]:5701 - 36651420-f623-4ac0-adc7-73326fba9fa8 this
    Member [10.5.167.11]:5701 - f41c88f0-4e7a-466d-897a-b222effe19d2
]