Search code examples
jenkinsamazon-ec2amazon-eksjenkins-agentjenkins-x

Jenkins on EKS can't connect to external EC2 server as Jenkins worker-node: ERROR `port:50000 is not reachable`

My main task is to create a Jenkins Node from an EC2 instance/server.

Jenkins server it's on EKS deployed via jx

$ jx version

jx                 2.0.818
jenkins x platform 2.0.1376
Kubernetes cluster v1.13.12
kubectl            v1.17.0
helm client        Client: v2.16.1
git                2.23.0
Operating System   Mac OS X 10.15.4

The problems I am facing:

  • The Helm chart for Jenkins deploys 2 services named jenkins-agent and jenkins. related issue
  • The ingress it's configured by default to use just jenkins service as backend and port 50000 isn't reachable for external agents via HOST.
  • I can't edit the jenkins ingress to point at service jenkins with 2 backend ports 8080 & 50000 (ingress allows just one port per service/path!). related issue

You can see here that here is 2 services: 1 jenkins for Jenkins controller and one for Jenkins agent jenkins-agent, but only one ingress jenkins which points to jenkins svc backend on port 8080.

$ kubectl get svc
NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)
heapster                    ClusterIP       ************    <none>         8082/TCP
jenkins                     ClusterIP       ************    <none>         8080/TCP
jenkins-agent               ClusterIP      ************     <none>         50000/TCP
jenkins-x-chartmuseum       ClusterIP      ************     <none>         8080/TCP
jenkins-x-docker-registry   ClusterIP     ************      <none>         5000/TCP



$ kubectl get ingress
NAME              HOSTS                             ADDRESS            PORTS
chartmuseum       chartmuseum.**                    ***.amazonaws.com   80  
docker-registry   docker-registry.**                ***.amazonaws.com   80
jenkins           jenkins.**                        ***.amazonaws.com   80

I tried to create another ingress specific for jenkins-agent svc and to use something like alb.ingress.kubernetes.io/group.name: mygroup. info link

I failed to bind 2 ingresses together, unfortunately the jenkins ingress has kubernetes.io/ingress.class: nginx and ingress.class does't have this feature...

I checked the SecurityGroups to have the port 50000 open.

I tried to go on AWS and to add manually a 50000 port to load balancer but still when I am running on EC2 this command:

java -jar agent.jar -jnlpUrl http://JENKINS_URL/computer/****-service/slave-agent.jnlp -secret ****** -workDir "/home/ec2-user/jenkins_home" -failIfWorkDirIsMissing

Fails with this Error:

provided port:50000 is not reachable

How to add an external Jenkins agent to a Jenkins which 2 services one for UI and Another for Agent. The above java command requires both ports to be open!

On the same PATH because you need port 8080 to get the slave-agent.jnlp and after you get it, in the background it's trying to connect to port 50000

I am out of ideas...

Solution

  • After experimenting all day...

    I came with a solution, it's not the best or straight forward but at least it's working. My slave is connected! INFO: Connected

    So, I edited the jenkins-agent service and from type: ClusterIP I changed to type: LoadBalancer. That gave me a new fresh loadbalancer and after I decided to add the port 8080 because in the service yaml file it's using the same selector as in jenkins service.

    (jenkins-agent service yaml)

      spec:
  clusterIP: ***
  externalTrafficPolicy: Cluster
  ports:
  - name: slavelistener
    nodePort: 30258
    port: 50000
    protocol: TCP
    targetPort: 50000
  - name: http
    nodePort: 30840
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    component: jenkins-x-jenkins-master
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: ****

    After I had just to add the port 8080 at the end of my new jenkins-agent load balancer:

    java -jar agent.jar -jnlpUrl http://****.elb.amazonaws.com:8080/computer/***-service/slave-agent.jnlp -secret ***** -workDir "/home/ec2-user/jenkins_home" -failIfWorkDirIsMissing

    INFO: Connected