I am testing Flink autoscaler with a Kubernetes setup using Flink Kubernetes Operator 1.5.0.
In the docs it says:
In the current state the autoscaler works best with Kafka sources, as they expose all the standardized metrics. It also comes with some additional benefits when using Kafka such as automatically detecting and limiting source max parallelism to the number of Kafka partitions.
How is that relevant? I mean, as I understand it scales based on % busy of each vertex.
Does that paragraph mean that sources other than Kafka may not report % busy, or that it can use Kafka lag metric to scale?
Ideally I'd like to scale based on Kafka lag, but I am not sure if this metric is available inside Flink
I would recommend to read FLIP-271 which introduced autoscaling to the Kubernetes Operator. See https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
The goal of the autoscaler algorithm is "to yield a resource efficient backpressure-free configuration in very few amount of scaling decisions." In order to achieve that, it uses multiple metrics, all of which are available out-of-the-box for the Flink Kafka connector.
Kafka lag itself isn't relevant to Flink. Flink only commits its offsets during snapshotting, to help with monitoring results in Kafka, but it doesn't need that for its fault tolerance. It also means that Kafka lag will increase until the moment Flink snapshots, but Flink has actually continued with reading messages from Kafka. It just hasn't committed the offsets yet.