I have deployed akka cluster in EKS and it is forming a cluster. I am making couple of rest calls to the cluster on a port 27520 and I am using cluster client to send it over to another akka cluster over clusterIP.
I see in the logs that it is getting the right service IP , but fails to communicate with the pods tied to the service IP.
I get the below error
initialContacts: Set(akka://[email protected]:8080/system/receptionist)
MktoClusterClientPoolActor: getClusterClientPool succeeded, setting retires to 1
[WARN ] [a.r.a.InboundHandshake$$anon$2] - Dropping Handshake Request from [akka://[email protected]:25520#-3378415211190166853] addressed to unknown local address [akka://[email protected]:8080]. Local address is [akka://[email protected]:25520]. Check that the sending system uses the same address to contact recipient system as defined in the 'akka.remote.artery.canonical.hostname' of the recipient system. The name of the ActorSystem must also match.
Service IP : 194.169.252.192
Pod IP: 101.65.16.86
Service port : 8080
Akka remoting port: 25520
I have defined service-settings.yaml where I associated ports:
ports:
- port: 8081
protocol: TCP
targetPort: 27520
name: http
- port: 8080
protocol: TCP
targetPort: 25520
name: remoting
Also, I can see under service in kubernetes dashboard that my ports are tied to it.I can port forward to my service locally and make a rest endpoint call that goes to one of the pods, but then the cluster client has difficulty in sending the request over to another akka cluster even after resolving the right IP address of the service.
I was going to the akka documentation but was confused whether it would help me : https://doc.akka.io/docs/akka/current/remoting-artery.html#akka-behind-nat-or-in-a-docker-container
Should I defined cononical host and port as my service host (controller) and port? and what should I defined by bind host and port as?
NOTE: I am aware cluster client is no longer supported and we should move to grpc. However, thats a lot of work which was not considered earlier
I could achieve the desired results (without switching to grpc) by directly connecting to pods. https://doc.akka.io/docs/akka-management/current/discovery/kubernetes.html. And refreshing the cluster client if the pod goes down.