Search code examples
spring-boothazelcastopenstack-novajclouds

jclouds connection refused during hazelcast clustering

Questions

  • Why does it use localhost?
  • What does keystone have to do with it?
    • I can't seem to configure a keystone endpoint

Context

  • App: Spring Boot (1.5.6) REST API
  • Hibernate 5.2
  • Hazelcast 3.9 - as 2nd-level cache only
  • hazelcast-jclouds 3.7.1
  • jclouds-compute and jclouds-allcompute 2.0.2
  • Openstack cloud for VMs running the app

The Setup

I have my hazelcast.xml configured as follows:

        <discovery-strategies>
            <discovery-strategy class="com.hazelcast.jclouds.JCloudsDiscoveryStrategy" enabled="true">
                <properties>
                    <property name="modules">org.jclouds.logging.slf4j.config.SLF4JLoggingModule</property>
                    <property name="provider">openstack-nova</property>
                    <property name="endpoint">http://dev.nova.cloud.youdontknow.net:8774/v2/</property>
                    <property name="identity">redacted</property>
                    <property name="credential">cens0red</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>

The problem

App initialization fails. Here's some log tidbits:

[TRACE] o.j.r.internal.RestAnnotationProcessor   : looking up default endpoint for org.jclouds.openstack.keystone.v2_0.AuthenticationApi.public abstract org.jclouds.openstack.keystone.v2_0.domain.Access org.jclouds.openstack.keystone.v2_0.AuthenticationApi.authenticateWithTenantNameAndCredentials(java.lang.String,org.jclouds.openstack.keystone.v2_0.domain.PasswordCredentials)[bnet-web, PasswordCredentials{username=redacted, password=*****}]
[TRACE] o.j.r.internal.RestAnnotationProcessor   : using default endpoint Optional.of(http://localhost:5000/v2.0/) for org.jclouds.openstack.keystone.v2_0.AuthenticationApi.public abstract org.jclouds.openstack.keystone.v2_0.domain.Access org.jclouds.openstack.keystone.v2_0.AuthenticationApi.authenticateWithTenantNameAndCredentials(java.lang.String,org.jclouds.openstack.keystone.v2_0.domain.PasswordCredentials)[bnet-web, PasswordCredentials{username=redacted, password=*****}]
[TRACE] o.j.rest.internal.InvokeHttpMethod       : << converted AuthenticationApi.authenticateWithTenantNameAndCredentials to POST http://localhost:5000/v2.0/tokens HTTP/1.1

And here's bits of the exception stack traces:

Caused by: com.hazelcast.core.HazelcastException: Failed to get registered addresses
at com.hazelcast.jclouds.JCloudsDiscoveryStrategy.discoverNodes(JCloudsDiscoveryStrategy.java:93)
at com.hazelcast.jclouds.JCloudsDiscoveryStrategy.discoverLocalMetadata(JCloudsDiscoveryStrategy.java:106)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:91)

...

Caused by: org.jclouds.http.HttpResponseException: Connection refused: connect connecting to POST http://localhost:5000/v2.0/tokens HTTP/1.1
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:122)
    ...
    at com.sun.proxy.$Proxy147.authenticateWithTenantNameAndCredentials(Unknown Source)
    at org.jclouds.openstack.keystone.v2_0.functions.AuthenticatePasswordCredentials.authenticateWithTenantName(AuthenticatePasswordCredentials.java:43)

Other Notes

Looks like it's using the default keystone address configured in org.jclouds.openstack.keystone.v2_0.KeystoneApiMetadata - but I don't know how that's involved.

Solution

  • By looking at the code, I think hazlecast-jclouds is not prepared to manage generic APIs. When connecting to a provider, you don't need to specify the endpoint, as it is well-known (the AWS endpoints, Google, Azure, etc), but when using generic APIs such as OpenStack or CloudStack, you need to tell jclouds where to connect. Unfortunately, it looks like hazlecast-jclouds lacks support for configuring custom endpoints for generic APIs.

    A quick look at the code suggests that it could be easy to add, though. The properties that are taken into account are defined in the JCloudsDiscoveryStrategyFactory, and then read in the ComputeServiceBuilder to create the jclouds context.

    I'm not familiar with Hazlecast, but I'd say that adding the definition for the "endpoint" property, and then, if present, configuring it by calling the jclouds contextBuilder.endpoint(endpoitn) method should do the trick.