Search code examples
apache-sparkapache-flink

How to add files to executor machines in Flink?

I'm trying to figure out ways to add some files to executors in Flink. Just like SparkContext.addFile() SparkFiles.get() in Spark, are there any possible APIs in Flink? Thanks.

Solution

  • You can check the documentation. The Scala API doc for ExecutionEnvironment class that serves as an entry point to Flink, has a registerCachedFile method. This can load a file from a distributed storage(for e.g hdfs) and make it available to all UDFs inside the code.

    This is the link to the page in the Doc stating the same Flink Doc link Check the section for Broadcast Variables