I have a Kubernetes deployment that deploys a pod that will pull down a single message from a RabbitMQ queue. I'm also using KEDA to scale the deployment based on the RabbitMQ messages currently in the queue. It correctly scales to 0, and then to 1 whenever there is a message, but the deployment never scales above 1. My current deployment YAML file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: scale
labels:
app: scale
spec:
replicas: 1
selector:
matchLabels:
app: scale
template:
metadata:
labels:
app: scale
spec:
containers:
- name: scale-deployment
image: bharper7/scale:v1
imagePullPolicy: Never
My KEDA YAML file:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: scale-keda-deployment
labels:
app: scale
deploymentName: scale
spec:
scaleTargetRef:
name: scale
pollingInterval: 5
minReplicaCount: 0
maxReplicaCount: 10
cooldownPeriod: 60
triggers:
- type: rabbitmq
metadata:
host: amqp://EmZn4ScuOPLEU1CGIsFKOaQSCQdjhzca:[email protected]:30861
mode: QueueLength
value: '1'
queueName: scaleTest
The KEDA operator log files:
2021-04-28T19:25:39.846Z INFO scaleexecutor Successfully updated ScaleTarget {"scaledobject.Name": "scale-keda-deployment", "scaledObject.Namespace": "default", "scaleTarget.Name": "scale", "Original Replicas Count": 0, "New Replicas Count": 1}
2021-04-28T19:25:40.272Z INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "default", "ScaledObject.Name": "scale-keda-deployment"}
I know everything as far as the RabbitMQ connection is working, and that KEDA knows what deployment to look at, as well as what queue. All of this is proven by the fact that the pod scales to 0 and 1. But for some reason it never goes beyond 1 even when I have 50 messages in the queue.
So far I've tried messing around with the pollingInterval and cooldownPeriod tags but neither seem to have an effect. Any ideas?
Edit:
I removed the replicas value from the deployment YAML file as suggested below. And also looked at the HPA logs.
The generated HPA logs:
Name: keda-hpa-scale-keda-deployment
Namespace: default
Labels: app=scale
app.kubernetes.io/managed-by=keda-operator
app.kubernetes.io/name=keda-hpa-scale-keda-deployment
app.kubernetes.io/part-of=scale-keda-deployment
app.kubernetes.io/version=2.1.0
deploymentName=scale
scaledObjectName=scale-keda-deployment
Annotations: <none>
CreationTimestamp: Wed, 28 Apr 2021 11:24:15 +0100
Reference: Deployment/scale
Metrics: ( current / target )
"rabbitmq-scaleTest" (target average value): 4 / 20
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from external metric rabbitmq-scaleTest(&LabelSelector{MatchLabels:map[string]string{scaledObjectName: scale-keda-deployment,},MatchExpressions:[]LabelSelectorRequirement{},})
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 22m (x6 over 23m) horizontal-pod-autoscaler deployments/scale.apps "scale" not found
Normal SuccessfulRescale 15m horizontal-pod-autoscaler New size: 1; reason: All metrics below target
This is after sending 5 messages into the queue. For some reason it only thinks 1 pod is needed even though I set value to 1 in the KEDA YAML file.
In case anyone else stumbles across this issue, I managed to find a work around. According the docs, the use of queueLength is deprecated and mode should be used instead. But changing back to the deprecated tags worked for me, for some reason the new tags don't work. Not really a proper fix but at least it got my deployment scaling as expected. My KEDA deployment file now looks like this:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: keda-deployment
labels:
apps: ffmpeg
deploymentName: ffmpeg
spec:
scaleTargetRef:
name: ffmpeg
pollingInterval: 10
cooldownPeriod: 1200
maxReplicaCount: 50
triggers:
- type: rabbitmq
metadata:
host: amqp://EmZn4ScuOPLEU1CGIsFKOaQSCQdjhzca:[email protected]:30861
queueName: files
queueLength: '1'