kuberneteskubernetes-health-checkmlrunigztop
k8s, without information about CPU and Memory
I got empty values for CPU and Memory, when I used igztop for check running pods in iguazio/mlrun solution. See the first line in output for this pod *m6vd9:
[ jist @ iguazio-system 07:41:43 ]->(0) ~ $ igztop -s cpu
+--------------------------------------------------------------+--------+------------+-----------+---------+-------------+-------------+
| NAME | CPU(m) | MEMORY(Mi) | NODE | STATUS | MLRun Proj. | MLRun Owner |
+--------------------------------------------------------------+--------+------------+-----------+---------+-------------+-------------+
| xxxxxxxxxxxxxxxx7445dfc774-m6vd9 | | | k8s-node3 | Running | | |
| xxxxxx-jupyter-55b565cc78-7bjfn | 27 | 480 | k8s-node1 | Running | | |
| nuclio-xxxxxxxxxxxxxxxxxxxxxxxxxx-756fcb7f74-h6ttk | 15 | 246 | k8s-node3 | Running | | |
| mlrun-db-7bc6bcf796-64nz7 | 13 | 717 | k8s-node2 | Running | | |
| xxxx-jupyter-c4cccdbd8-slhlx | 10 | 79 | k8s-node1 | Running | | |
| v3io-webapi-scj4h | 8 | 1817 | k8s-node2 | Running | | |
| v3io-webapi-56g4d | 8 | 1827 | k8s-node1 | Running | | |
| spark-worker-8d877878c-ts2t7 | 8 | 431 | k8s-node1 | Running | | |
| provazio-controller-644f5784bf-htcdk | 8 | 34 | k8s-node1 | Running | | |
and It also was not possible to see performance metrics (CPU, Memory, I/O) for this pod in Grafana.
Do you know, how can I resolve this issue without whole node restart (and what is the root cause)?
Solution
It seems as the issue with kubelet, the best is to follow the next step by step scenario (see diagram in pdf)
- Configure Microk8s
- How do I get a certificate (public and private key) into a windows container in AKS?
- "The connection to the server localhost:8080 was refused - did you specify the right host or port?"
- 1 node(s) had taints that the pod didn't tolerate in kubernetes cluster
- --record has been deprecated, then what is the alternative
- Flink Kubernetes S3 state support
- How to count how many pods were started by namespace each day in PromQL?
- couldn't get current server API group list: the server has asked for the client to provide credentials error: You must be logged in to the server
- access postgres in kubernetes from an application outside the cluster
- What will happen to evicted pods in kubernetes?
- Mongo DB deployment not working in kubernetes because processor doesn't have AVX support
- Flux not decrypting using SOPS
- Python Kubernetes Client: equivalent of kubectl api-resources --namespaced=false
- Not possible to use containers in Azure Pipelines self-hosted agents deployed as Azure K8s pods
- My kubernetes pods keep crashing with "CrashLoopBackOff" but I can't find any log
- GKE autoscaling
- how do I enable mount propagation in Rancher - Kubernetes feature gates?
- How to resolve unmet dependencies between containerd and runc?
- Shell (ssh) into Azure AKS (Kubernetes) cluster worker node
- How to enable Calico network policy on the existing AKS cluster
- Azure Kubernetes Service (AKS) uses Application Gateway Ingress Controller (AGIC) : How to implement HSTS header in ASP.Net Core 6.0?
- TLS Termination in K8S Gateway API (azure implementation) Resource?
- Any python client method available to get/set env variables of kubernetes deployment
- In Kubernetes, how can I scale a Deployment to zero when idle
- ClusterIssuer Failed to register ACME account: secret already exists
- kubectl - How to restart a deployment (or all deployment)
- Docker Desktop 1node Kubernetes Jenkins share docker.sock from host into agent docker container
- How can I get the prometheus metrics for informer watch operation latency?
- How to manage persistent connections in kubernetes
- Port 6443 connection refused when setting up kubernetes