WARN i.v.c.e.i.clustered.ConnectionHolder failed

Question

I get an error message:

10:29:56.116 [vert.x-worker-thread-18] WARN  i.v.c.e.i.clustered.ConnectionHolder - Connecting to server localhost:53990 failed

for which I found the discussion: https://groups.google.com/forum/#!topic/vertx/bws3x9-WsV0 where one reply was:

How do you start Vert.x? There's a hostname detection mechanism in the CLI and the Launcher class. If you have your own main class, make sure to set the ClusterHost option correctly.

So i looked for the description in https://vertx.io/docs/vertx-hazelcast/java/

Which says:

When running Vert.x is in clustered mode, you should also make sure that Vert.x knows about the correct interface. When running at the command line this is done by specifying the cluster-host option: vertx run myverticle.js -cluster -cluster-host your-ip-address Where your-ip-address is the same IP address you specified in the Hazelcast configuration. If using Vert.x programmatically you can specify this using setClusterHost.

I don'get this since I was assuming that localhost would be just fine. I am just trying a two node configuration see https://github.com/rc-dukes/dukes/issues/19

What is needed to understand and fix this?

https://github.com/vert-x3/vertx-maven-starter/issues/6

show a scenario where another starter might mess up things. How could i debug this?

Answer 1

The log output shows that the two vert.x clients involved have indeed properly connected via the Multicast Joiner (x.y.z is hidding the real ip):

08:42:09.539 [hz._hzInstance_1_dev.priority-generic-operation.thread-0] INFO  c.h.internal.cluster.ClusterService - [x.y.z.82]:5701 [dev] [3.12.5] 

Members {size:2, ver:8} [
    Member [x.y.z.82]:5701 - 29b35836-aad1-4314-af62-c453ce200ffa this
    Member [x.y.z.25]:5701 - f67995b7-f47e-4037-b7a4-fcedeff4fc91
]

One of the clients is a raspberry PI and per default it identifies itself with a 127.0.0.1 host address.

Removing that entry in /etc/hosts and replacing it with a proper entry with full ip address is the first necessary step.

cat /etc/hosts
127.0.0.1   localhost
::1     localhost ip6-localhost ip6-loopback
ff02::1     ip6-allnodes
ff02::2     ip6-allrouters

# on the main network
x.y.z.82 picarford.bitplan.com picarford

Then the hostname needs to be used as host/clusterhost for all members.

hostname=InetAddress.getLocalHost().getCanonicalHostName()
starter.setClusterHost(hostname,hostname)

see ClusterStarter.java

   /**
   * prepare the starter
   */
  public void prepare() {
    if (!prepared) {
      Config.configureLogging();
      try {
        setHostname(InetAddress.getLocalHost().getCanonicalHostName());
      } catch (UnknownHostException e) {
        LOG.error(e.getMessage());
      }
      prepared = true;
    }
  }

  private String hostname;

  /**
   * configure the cluster
   * @param clusterHostname
   * @param publicHost
   */
  public void configureCluster(String clusterHostname,String publicHost) {
    if (clusterHostname == null) {
      clusterHostname=getHostname();
    }
    if (publicHost==null) {
      publicHost=getHostname();
    }
    String msg=String.format("starting cluster on %s setting host to %s and clusterPublicHost to %s",getHostname(),clusterHostname,publicHost);
    LOG.info(msg);
    EventBusOptions eventBusOptions = getOptions().getEventBusOptions();
    // https://github.com/eclipse-vertx/vert.x/issues/3229
    // https://stackoverflow.com/a/49028531/1497139
    // https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBusOptions.html#setClusterPublicHost-java.lang.String-
    eventBusOptions.setHost(clusterHostname);
    eventBusOptions.setClusterPublicHost(publicHost);
  }