Search code examples
apache-flinkhadoop-yarnflink-streaming

Flink configurations to avoid uploading extenal jars every time submit job

Does Flink has configurations to avoid uploading extenal jars every time submit job, like spark.yarn.archive in spark-submit?

I know ways to deal with external jars like answers in: How to Reference the External Jar in Flink. Such as fat jar, Copying jars to $FLINK/lib, -yt configuration, but they are not flexible or convenient enough.

Solution

  • I notice Flink has an issue FLINK-13938 to resolve this, and the improve haved been merged in recent version 1.11.0.

    A final submission command could be issued like following.

    ./bin/flink run -m yarn-cluster -d \
-yD yarn.provided.lib.dirs=hdfs://myhdfs/flink/lib,hdfs://myhdfs/flink/plugins \
examples/streaming/WindowJoin.jar