I'm running Jenkins on Kubernetes with a dynamic slave pod provisioning setup.
It used to work well for a long time but somehow started to giving this issue from yesterday.
This is the log that I'm getting from the slave pods
Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: default-jxrsk
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jun 26, 2021 10:48:16 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.40
Jun 26, 2021 10:48:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/remoting as a remoting work directory
Jun 26, 2021 10:48:16 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/remoting
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins.jenkins:8080]
Jun 26, 2021 10:48:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jun 26, 2021 10:48:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-agent
Agent port: 50000
Identity: 04:b1:b3:70:fe:e4:89:1b:30:62:4d:c9:6a:53:8a:3b
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent:50000
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 04:b1:b3:70:fe:e4:89:1b:30:62:4d:c9:6a:53:8a:3b
Jun 26, 2021 10:48:16 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to jenkins-agent/10.245.115.232:50000] Local headers refused by remote: Unknown client name: default-jxrsk
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: default-jxrsk
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:579)
at hudson.remoting.Engine.run(Engine.java:490)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: default-jxrsk
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
... 7 more
Jun 26, 2021 10:48:16 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:664)
at hudson.remoting.Engine.innerRun(Engine.java:604)
at hudson.remoting.Engine.run(Engine.java:490)
And getting this from the master Jenkins
2021-06-26 10:47:57.232+0000 [id=73] INFO o.j.r.p.i.ConnectionHeadersFilterLayer#onRecv: [JNLP4-connect connection from 10.244.75.115/10.244.75.115:48028] Refusing headers from remote: Unknown client name: default-t09rx
2021-06-26 10:48:01.652+0000 [id=29] INFO o.c.j.p.k.KubernetesCloud#provision: Excess workload after pending Kubernetes agents: 1
2021-06-26 10:48:01.653+0000 [id=29] INFO o.c.j.p.k.KubernetesCloud#provision: Template for label null: default
2021-06-26 10:48:11.659+0000 [id=73] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: jenkins/default-jxrsk
2021-06-26 10:48:11.660+0000 [id=73] WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: default-jxrsk, template=PodTemplate{inheritFrom='', name='default', namespace='', hostNetwork=false, label='jenkins-jenkins-slave', serviceAccount='default', nodeSelector='doks.digitalocean.com/node-pool=k8s-generic', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='myrepo/myimage:jenkins_jnlp-3.40-1-helm3', alwaysPullImage=true, workingDir='/home/jenkins', command='', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='1000m', resourceRequestMemory='4096Mi', resourceLimitCpu='1000m', resourceLimitMemory='4096Mi', envVars=[ContainerEnvVar [getValue()=http://jenkins.jenkins:8080, getKey()=JENKINS_URL], KeyValueEnvVar [getValue()=jenkins-agent:50000, getKey()=JENKINS_TUNNEL]], livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@2e38f1da}], imagePullSecrets=[org.csanchez.jenkins.plugins.kubernetes.PodImagePullSecret@5a6fc800]}
java.lang.NoSuchMethodError: io.fabric8.kubernetes.client.dsl.PodResource.watch(Ljava/lang/Object;)Ljava/lang/Object;
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:134)
at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-06-26 10:48:11.661+0000 [id=73] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent default-jxrsk
Any idea? I was trying to Google it for a whole day, but didn't find anything really helpful.
Need your help guys. Thanks.
Self reply:
I was able to provision slave pods again after I upgraded my Jenkins and Kubernetes plugin to the latest.
The version I'm at is Jenkins 2.289.1, and Kubernetes plugin 1.30.0.