How can I package a Flink job along with a Java application and run it as a Docker image using docker-compose and a fat jar?

I am working with Flink using docker compose. I want to package together the flink jobmanager, taskmanager and one java application that I have.

Currently I am aware that I can package the java application in a fat jar and I can upload it using the web ui of Flink, however I've been asked if it is possible to have a script that "builds" the fat-jar into flink in a way that it creates a docker image and whenever it is started the job is there running.

As explained I can do all of this manually step by step but the goal is to automate this as we might have more java applications and at this moment we do not have a flink cluster so they want to package together per application.

I have tried uploading the fat jar from web ui, uploading the fat jar using the rest api, using a flink deployer that I found but I have not managed to automate it

Solution

After a few days of doing research and trial and error I've found the following solution to my problem:

In the Java app project, I created a Dockerfile:

FROM gradle:7.3.3 as gradleBuild
COPY --chown=gradle:gradle . /home/gradle/src
WORKDIR /home/gradle/src
RUN gradle clean build --no-daemon

FROM flink:1.16.1
COPY --from=gradleBuild /home/gradle/src/build/libs/*all.jar /opt/flink/usrlib/artifact

Basically with the docker file we are copying the source files of our Java project into the gradle image and running a clean build, once the project is compiled, it generates a normal jar and a fat jar (I am using a gradle plugin to generate fat jar automatically), we copy the fat jar into Flink in usrlib directory and this generates a new image of Flink including our Java app.

Then I created the following docker-compose file:

version: '3.4'
services: 
  jarbuild:
    build: .
    image: flinkjar
  
  jobmanager:
    container_name: jobmanager_jar
    image: flinkjar
    depends_on:
      - jarbuild
    ports:
      - "8081:8081"
      - "6123:6123"
    command: standalone-job --job-classname org.example.Classname #Here you put the classname for your job
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.adress: jobmanager
        jobmanager.web.log.path: /opt/flink/log/output.log
        parallelism.default: 5  #our job can run in parallel

   taskmanager:
     container_name: taskmanager_jar
     image: flinkjar
     depends_on:
       - jobmanager
     command: taskmanager
     scale: 1
     environment:
       - |
         FLINK_PROPERTIES=
         jobmanager.rpc.address: jobmanager
         taskmanager.numberOfTaskSlots: 6
         taskmanager.web.log.path: /opt/flink/log/output.log
         parallelism.default: 5

This is very similar to what you can find in the apache flink notes to run it using docker compose, the only difference is the command we are using in the jobmanager, specifying this will be a standalone job and passing the name of the class of our Java app, since this is included in the usrlib directory, flink will start the job as soon as the jobmanager and taskmanager start.

The next step is to run:

docker-compose build

After the image is created now you can run:

docker-compose up

Be aware that if you make changes to your Java application, it is required that you run docker-compose build again, running docker-compose up will use an existing image if it is there without building a new one with the changes.