I've a namespace I'm unable to delete in my Kubernetes cluster. When I run kubectl get ns traefik -o yaml, I get the following:
apiVersion: v1
kind: Namespace
metadata:
annotations:
cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:28:59Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:29:00Z"}]}'
field.cattle.io/projectId: c-5g2hz:p-bl9sf
lifecycle.cattle.io/create.namespace-auth: "true"
creationTimestamp: "2021-06-11T20:28:58Z"
deletionTimestamp: "2021-07-04T07:21:20Z"
labels:
field.cattle.io/projectId: p-bl9sf
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:field.cattle.io/projectId: {}
f:labels:
.: {}
f:field.cattle.io/projectId: {}
f:status:
f:phase: {}
manager: agent
operation: Update
time: "2021-06-11T20:28:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cattle.io/status: {}
f:lifecycle.cattle.io/create.namespace-auth: {}
manager: rancher
operation: Update
time: "2021-06-11T20:28:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
.: {}
k:{"type":"NamespaceContentRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionContentFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionDiscoveryFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionGroupVersionParsingFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceFinalizersRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
manager: kube-controller-manager
operation: Update
time: "2021-07-04T07:21:26Z"
name: traefik
resourceVersion: "15400692"
uid: 4b198956-bbd5-4bdb-9dc6-9d53feda91e4
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2021-07-04T07:21:25Z"
message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
unable to handle the request'
reason: DiscoveryFailed
status: "True"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content successfully removed
reason: ContentRemoved
status: "False"
type: NamespaceContentRemaining
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content-preserving finalizers finished
reason: ContentHasNoFinalizers
status: "False"
type: NamespaceFinalizersRemaining
phase: Terminating
And when I run kubectl delete ns traefik --v=10, the last output is the following:
I0708 18:38:26.538676 31537 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" 'http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true'
I0708 18:38:27.013394 31537 round_trippers.go:445] GET http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true 200 OK in 474 milliseconds
I0708 18:38:27.013421 31537 round_trippers.go:451] Response Headers:
I0708 18:38:27.013427 31537 round_trippers.go:454] Access-Control-Allow-Origin: *
I0708 18:38:27.013450 31537 round_trippers.go:454] Date: Thu, 08 Jul 2021 16:38:27 GMT
I0708 18:38:27.013453 31537 round_trippers.go:454] Connection: keep-alive
I0708 18:38:27.013468 31537 request.go:708] Unexpected content type from the server: "": mime: no media type
I already tried to remove the finalizers as described on https://www.ibm.com/docs/en/cloud-private/3.2.0?topic=console-namespace-is-stuck-in-terminating-state but after some seconds I finally get EOF:
> curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/traefik/finalize
EOF
Is anyone having any idea how I can delete the traefik namespace?
Posting this as a community wiki out of comments, feel free to edit and expand.
After analysing a state of the problematic namespace, this part was the main cause of the issue:
message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
unable to handle the request'
The issue was with metric server in kubernetes. Once the metric server is available, namespace can be unblocked to be deleted.
Similar issue was resolved in another answer on the stackoverflow.