Search code examples
javaapache-flink

Flink : Load resource file in cluster mode


I am running a Flink code that work well when executed localy (in IntelliJ) but crash when running in a Flink cluster :

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.

In the initialisation of my job, I load a text file, that is embedded in my jar file with the following code :

try (InputStream is = new FileInputStream(new File("src/main/resources/my_file.txt"))) {
            myObject = new MyClass.Parser().parse(is);
}catch (final Exception e) {
            LOGGER.error("Error while trying to file : "+e.getMessage(), e);
}

The object created from the content of the text file is then used to create my Source and my Sink and also to do some internal computing in a map function.

How can I correctly load this file in cluster mode ?

Edit :

When I replace the code above with

myObject = new MyClassParser().parse(FILE_CONTENT_AS_STRING);

The code run well. It looks like it is only the loading of the file that cause the error.


Solution

  • Once you build your code into a jar, the file path no longer can be used to access the file. Instead, you need to use something like <YourClassName>.class.getResourceAsStream("/my_file.txt") to get the InputStream.