Kubernetes etcd HighNumberOfFailedHTTPRequests QGET
I run kubernetes cluster in AWS, CoreOS-stable-1745.6.0-hvm (ami-401f5e38), all deployed by kops 1.9.1 / terraform.
etcd_version = "3.2.17"
k8s_version = "1.10.2"
This Prometheus alert method=QGET alertname=HighNumberOfFailedHTTPRequests is coming from coreos kube-prometheus monitoring bundle. The alert started to fire from the very beginning of the cluster lifetime and now exists for ~3 weeks without visible impact.
^ QGET fails - 33% requests.
NOTE: I have the 2nd cluster in other region built from scratch on the same versions and it has exact same behavior. So it's reproducible.
Anyone knows what might be the root cause, and what's the impact if ignored further?
EDIT: Later I found this GH issue which describes my case precisely: https://github.com/coreos/etcd/issues/9596
I got the explanation in GitHub issue thread. HTTP metrics/alerts should be replaced with GRPC.
- Waiting on certificate issuance from order status "pending"
- Unable to use s3-fs-hadoop plugin in Kubernetes
- custom tag on EBS volume provisioned dynamically by Kubernetes
- How to duplicate a specific object in a specific overlay from base with kustomize?
- How to uninstall ArgoCD from Kubernetes cluster?
- Pod limit on Node - AWS EKS
- Terraform: Error: Kubernetes cluster unreachable: invalid configuration
- Kubernetes - what happens if you don't set a pod CPU request or limit?
- How do I navigate to next filter/search item in k9s?
- How to create aws-ebs-csi-driver with eks_blueprints_addons by Terraform?
- Your current user or role does not have access to Kubernetes objects on this EKS cluster
- During uvicorn startup child process dies in Kubernetes cluster
- How to create ConfigMap yaml manifest for Azure Kubernetes using shell or PowerShell script?
- How to add X-Original-Forwarded-For as request header in NGINX Ingress Controller?
- Are Kubernetes pods that use host networking assigned a pod IP that can be accessed using a Service resource?
- Can't access ArgoCD UI that is in a VM with port forwarding set in vagrant file
- How to enable ReadWriteMany access mode using an io2 EBS Volume
- How the multiple replicas of k8s deployment running on different node read from aws EBS volume in READWRITEMANY pv mode?
- Kubernetes PVC with ReadWriteMany on AWS
- Exclude Resource in kustomization.yaml
- Will HPA scale downs happen if clients are currently connected via web sockets?
- Installing Jenkins on Kubernetes behind corporate proxy
- Is there a kubernetes role definition to allow the command `kubectl rollout restart deploy <deployment>`?
- Kubernetes - delete all jobs in bulk
- Kubernetes Ingress non-root path 404 Not Found
- Why I have to specify - sh - -c - command instead of just - command in k8s livenessProbe exec command (and potentially elsewhere)
- How to automatically update secrets mounted as volumes?
- Kubectl error: memcache.go:265] couldn’t get current server API group list: Get
- How to patch configmap in kubernetes which has xml within yaml?
- How to make request to nginx server_name in kubernetes