Search code examples
apache-flinkshadowjarfatjar

Max size of a fat JAR for Apache Flink


I've build an Apache Flink app and packaged it in a fat JAR with Gradle Shadow Plugin. The resulting file size is ~114 MiB. When I'm trying to upload it with Flink's web UI it is stuck in "Saving…" phase. And if I use curl to upload it manually the result is "413 Request Entity Too Large":

$ curl -X POST -H "Expect:" -i -F "[email protected]" http://ec2-18-204-247-166.compute-1.amazonaws.com:8081/jars/upload

HTTP/1.1 413 Request Entity Too Large
content-length: 0

What are the options, then?

UPD: I can see the JAR in /tmp/flink-web-UUID/flink-web-upload/UUID/flink-all.jar but it is not recognized by Flink (not visible on UI).


Solution

  • Ok, it was easy to fix.

    First, I've scanned their repo for "Too Large" string and found this class. Looks like SERVER_MAX_CONTENT_LENGTH is responsible for max object size. It is set here from the configuration option rest.server.max-content-length. The default is 100 MiB.

    TLDR:

    Setting rest.server.max-content-length in flink-conf.yaml to 209715200 (200 MiB) solved the issue.