Deployment issue with K3S from GitLab Runner: "the server could not find the requested resource"

Question

I'm facing an issue when trying to deploy Kubernetes resources on a K3S cluster from a GitLab Runner. The error message indicates that the server cannot find the requested resource. Here are the relevant details:

• GitLab CI/CD is configured to deploy Kubernetes resources.
• The cluster is a K3S cluster.
• The kubectl apply command is used, and the error occurs during the API server request.
• I can successfully retrieve information from the cluster using kubectl get po -n kube-system.
• The GitLab Runner has the necessary permissions, as it can fetch pods from the cluster.

Here is the relevant portion of my .gitlab-ci.yaml file:

deploy:
  stage: deploy
  image:
    name: harbor.localdomain.ma/devops/lachlanevenson/k8s-kubectl:v1.9.2
    entrypoint: []
  extends:
    - .tag-small
  script:
    - mkdir $HOME/.kube
    - "echo $KUBECONF_DEFAULT | base64 -d > $HOME/.kube/config"
    - kubectl version --insecure-skip-tls-verify
    - kubectl get po -n kube-system 
    - pwd 
    - ls -lrth kubernetes
    - echo $CI_PROJECT_URL
    - cat /builds/BS-ocps/deployments/kubernetes/deployment.yaml > test.yaml
    - kubectl apply -f test.yaml -n bs -v=8 --insecure-skip-tls-verify
    - kubectl apply -f /builds/BS-ocps/deployments/kubernetes/service.yaml -n bs -v=8 --insecure-skip-tls-verify
    - kubectl get po -n bs

the error I get:

I1227 14:18:35.820599      57 loader.go:357] Config loaded from file /root/.kube/config
I1227 14:18:35.873465      57 round_trippers.go:414] GET https://CLUSTER_IP:6443/swagger-2.0.0.pb-v1
I1227 14:18:35.873550      57 round_trippers.go:421] Request Headers:
I1227 14:18:35.873567      57 round_trippers.go:424]     Accept: application/json, */*
I1227 14:18:35.873578      57 round_trippers.go:424]     User-Agent: kubectl/v1.9.2 (linux/amd64) kubernetes/5fa2db2
I1227 14:18:35.906258      57 round_trippers.go:439] Response Status: 404 Not Found in 32 milliseconds
I1227 14:18:35.906318      57 round_trippers.go:442] Response Headers:
I1227 14:18:35.906331      57 round_trippers.go:445]     X-Content-Type-Options: nosniff
I1227 14:18:35.906341      57 round_trippers.go:445]     X-Kubernetes-Pf-Flowschema-Uid: 3a3edf31-1f82-4b20-a18b-daed7dae0d7d
I1227 14:18:35.906393      57 round_trippers.go:445]     X-Kubernetes-Pf-Prioritylevel-Uid: a583e898-18a5-4e9d-9953-97a43f37688e
I1227 14:18:35.906403      57 round_trippers.go:445]     Content-Length: 19
I1227 14:18:35.906412      57 round_trippers.go:445]     Date: Wed, 27 Dec 2023 14:18:43 GMT
I1227 14:18:35.906422      57 round_trippers.go:445]     Audit-Id: 58cfcb04-f093-44b4-b3a5-b1c4d42f8816
I1227 14:18:35.906432      57 round_trippers.go:445]     Cache-Control: no-cache, private
I1227 14:18:35.906441      57 round_trippers.go:445]     Content-Type: text/plain; charset=utf-8
I1227 14:18:35.909213      57 request.go:873] Response Body: 404 page not found
I1227 14:18:35.912075      57 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server could not find the requested resource",
  "reason": "NotFound",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "404 page not found"
      }
    ]
  },
  "code": 404
}]
F1227 14:18:35.912159      57 helpers.go:119] Error from server (NotFound): the server could not find the requested resource
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 255

Any insights or suggestions on resolving this issue would be greatly appreciated. Thanks in advance!

I tried running the deployment.yaml on the cluster from my machine and it works, so there is nothing wrong with it.

Answer 1

I had this error because I was working with an outdated kubectl image in the runner, when updating to the latest one everything worked.