I have got a single jar that uses main function to initiate 3 different jobs (topologies) and I want to deploy it in a kubernetes cluster with flink kubernetes operator (version 1.4) in application mode. I would expect that all 3 jobs to be initiated as I was able to do it using native kubernetes deployment mode (link).
Instead only the first job is scheduled from my application, and other jobs are ignored. The simplified manifest used for FlinkDeployment resource is the following
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: my-flink-app
spec:
image: my-custom-flink-image-with-app
flinkVersion: v1_16
flinkConfiguration:
taskmanager.numberOfTaskSlots: "4"
serviceAccount: flink
jobManager:
resource:
memory: "1024m"
cpu: 1
taskManager:
resource:
memory: "1024m"
cpu: 1
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: task-manager-pod-template
spec:
initContainers:
- name: busybox
image: busybox:1.35.0
command: [ 'sh','-c','echo hello from task manager' ]
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: FLINK_PARALLELISM
value: "2"
job:
jarURI: local:////opt/flink/usrlib/my-app.jar
parallelism: 2
Is this behavior the expected one ? Am I missing something, shouldn't be possible to start a Flink application with multiple jobs using flink-kubernetes-operator ?
According to official documentation it is allowed on specific deployment mode.
Application deployments manage a single job deployment in Application mode while Session deployments manage Flink Session clusters without providing any job management for it. The type of cluster created depends on the spec provided by the user.