Search code examples
azure-aksignite

Failed to get registered addresses from IP finder error in my Apache Ignite cluster which is deployed in AKS


I am spinning up an ignite cluster in AKS using official documentation https://ignite.apache.org/docs/latest/installation/kubernetes/azure-deployment

I am using the below files to create the resources

service.yaml

apiVersion: v1
kind: Service
metadata:
  # The name must be equal to KubernetesConnectionConfiguration.serviceName
  name: ignite-service
  # The name must be equal to KubernetesConnectionConfiguration.namespace
  namespace: ignite
  labels:
    app: ignite
spec:
  type: LoadBalancer
  ports:
    - name: rest
      port: 8080
      targetPort: 8080
    - name: thinclients
      port: 10800
      targetPort: 10800
  # Optional - remove 'sessionAffinity' property if the cluster
  # and applications are deployed within Kubernetes
  #  sessionAffinity: ClientIP
  selector:
    # Must be equal to the label set for pods.
    app: ignite
status:
  loadBalancer: {}

created service account with cmd kubectl create sa ignite -n ignite

cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ignite
  namespace: ignite
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ignite
roleRef:
  kind: ClusterRole
  name: ignite
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: ignite
  namespace: ignite

deployment.yaml

# An example of a Kubernetes configuration for pod deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
  # Cluster name.
  name: ignite-cluster
  namespace: ignite
spec:
  # The initial number of pods to be started by Kubernetes.
  replicas: 2
  selector:
    matchLabels:
      app: ignite
  template:
    metadata:
      labels:
        app: ignite
    spec:
      serviceAccountName: ignite
      terminationGracePeriodSeconds: 100000
      containers:
        # Custom pod name.
      - name: ignite-node
        image: apacheignite/ignite:2.14.0
        env:
        - name: OPTION_LIBS
          value: ignite-kubernetes,ignite-rest-http
        - name: CONFIG_URI
          value: file:///ignite/config/node-configuration.xml
        ports:
        # Ports to open.
        - containerPort: 47100 # communication SPI port
        - containerPort: 47500 # discovery SPI port
        - containerPort: 49112 # dafault JMX port
        - containerPort: 10800 # thin clients/JDBC driver port
        - containerPort: 8080 # REST API
        volumeMounts:
        - mountPath: /ignite/config
          name: config-vol
      volumes:
      - name: config-vol
        configMap:
          name: ignite-config

node-configuration.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
                        <property name="namespace" value="ignite"/>
                        <property name="serviceName" value="ignite-service"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

created configmap using cmd kubectl create configmap ignite-config -n ignite --from-file=node-configuration.xml

the pods are up and running. However I am getting below error


[07:22:28,352][SEVERE][main][TcpDiscoverySpi] Failed to get registered addresses from IP finder (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries) [maxTimeout=0]
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:81)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:2057)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1992)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1283)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1120)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:472)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2212)
        at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278)
        at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:1089)
        at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1766)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1147)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1757)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1679)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1121)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1015)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:840)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:710)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:679)
        at org.apache.ignite.Ignition.start(Ignition.java:353)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: class org.apache.ignite.IgniteException: Failed to retrieve Ignite pods IP addresses.
        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:123)
        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:76)
        ... 20 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1572)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:111)
        ... 21 more

I see that the public IP has been assigned to the ignite-service loadbalancer. can anyone please help me with the issue.

Thanks in advance!!

I have tried a couple solutions in the net but they did not get me anywhere.


Solution

  • It looks to be related to a base image change in Apache Ignite starting from 2.14.

    I suggest the following: