Search code examples
google-compute-enginegoogle-cloud-platformkubernetesgcloudautoscaling

Kubernetes pod autoscaling out of sync with Instance Group autoscaling


I have a simple wordpress site defined by the ReplicationController and Service below. Once the app is deployed and running happily, I enabled autoscaling on the instance group created by Kubernetes by going to the GCE console and enabling autoscaling with the same settings (max 5, cpu 10).

Autoscaling the instances and the pods seem to work decent enough except that they keep going out of sync with each other. The RC autoscaling removes the pods from the CE instances but nothing happens with the instances so they start failing requests until the LB health check fails and removes them.

Is there a way to make kubernetes scale the pods AS WELL as scale the instances that they run on so this doesn't happen? Or is there a way to keep them in sync?

My process is as follows:

Create the cluster

$ gcloud container clusters create wordpress -z us-central1-c -m f1-micro

Create the rc

$ kubectl create -f rc.yml

Create the service

$ kubectl create -f service.yml

Autoscale the rc

$ kubectl autoscale rc frontend --max 5 --cpu-percent=10

Then I enabled the autoscaling in the console and gave the servers load to make them scale.

rc.yml

apiVersion: v1
kind: ReplicationController
metadata:
  name: frontend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - image: custom-wordpress-image
        name: wordpress
        ports:
          - containerPort: 80
            hostPort: 80

service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    name: frontend
  name: frontend
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    name: wordpress

Update for more information

If I don't use kubernetes autoscaler and instead set the replicas to the same number as the instance group autoscaler max instance count, I seem to get the desired result. As instances are added to the instance group, kubernetes provisions them, as they are removed kubernetes updates accordingly. At this point I wonder what the purpose of the Kubernetes autoscaler is for.


Solution

  • With the new addition of Kubernetes 1.3 autoscaling I can now have Kubernetes autoscale my cluster and my pods.

    Using GCP's create command I can now easily add an autoscaled cluster using the --enable-autoscaling combined with the --min-nodes, --max-nodes, and --num-nodes.