We recently moved to GKE autopilot knowing its more secure, stable and has less management overhead in terms of managing nodepools. we are seeing this as expected, but the biggest issue we saw is our cost going up 2-3 times. The reason is every container in every pod consumes 0.5 CPU an 512 MB Memory, which is minimum default for Balanced compute class. For most of our apps its set to 100 vcpu and 128MB as request and limits CAN go higher. now I know we cannot set different request limits in autopilot but using lesser resources was something we were looking for.
After researching a bit I saw we can use nodeSelectors on a pod and specify compute-class.
Performance compute class seems to be the one for my needs as it allows 1m CPU and 1 Mb memory at minimum.
for a given pod I tried setting it as
nodeSelector:
cloud.google.com/compute-class: Performance
After this we started seeing this error.
Failed to save resource: admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autopilot-compute-class-limitation]":["the specified 'cloud.google.com/compute-class:Performance' is not supported. Deployment 'XXX.mysvc'."]}
Requested by user: 'XXXXXXX', groups: 'system:authenticated'.
After researching further, I stumbled upon this link. https://cloud.google.com/kubernetes-engine/docs/how-to/performance-pods As explained we tried this example:
apiVersion: v1
kind: Pod
metadata:
name: performance-pod
spec:
nodeSelector:
cloud.google.com/compute-class: Performance
cloud.google.com/machine-family: c3
cloud.google.com/gke-ephemeral-storage-local-ssd: "true"
containers:
- name: my-container
image: "k8s.gcr.io/pause"
resources:
requests:
cpu: 100
memory: "128Mi"
ephemeral: "1Gi"
Then I saw this error.
Violations details: {"[denied by autogke-node-affinity-selector-limitation]":["Key 'cloud.google.com/machine-family' is not allowed with node selector; Autopilot only allows labels with keys: cloud.google.com/compute-class,cloud.google.com/gke-spot,cloud.google.com/gke-placement-group,topology.kubernetes.io/region,topology.kubernetes.io/zone,failure-domain.beta.kubernetes.io/region,failure-domain.beta.kubernetes.io/zone,cloud.google.com/gke-os-distribution,kubernetes.io/os,kubernetes.io/arch,cloud.google.com/private-node,sandbox.gke.io/runtime,cloud.google.com/gke-accelerator,cloud.google.com/gke-accelerator-count,iam.gke.io/gke-metadata-server-enabled."],"[denied by autopilot-compute-class-limitation]":["the specified 'cloud.google.com/compute-class:Performance' is not supported. Deployment 'xxx.xxxx'."]}
Requested by user: 'xxxx', groups: 'system:authenticated'.
now my question is, If I want to deploy a pod which has multiple containers and They should be able to request 128MB and 100m CPU. Can I do that using ANY compute class in GKE autopilot. If yes, any example or links would be much appreciated. Thank you.
If I want to deploy a pod which has multiple containers and They should be able to request 128MB and 100m CPU. Can I do that using ANY compute class in GKE autopilot. If yes, any example or links would be much appreciated
It should be possible to set the request with General-purpose class with bursting. The minimum that you can request with bursting is 50m CPU and 52 MiB Memory for General-purpose.
Note that the General-purpose class is the default Compute type and this is automatically setup if you didn't specify the nodeSelector in the deployment.