I am running a Flink code that work well when executed localy (in IntelliJ) but crash when running in a Flink cluster :
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
In the initialisation of my job, I load a text file, that is embedded in my jar file with the following code :
try (InputStream is = new FileInputStream(new File("src/main/resources/my_file.txt"))) {
myObject = new MyClass.Parser().parse(is);
}catch (final Exception e) {
LOGGER.error("Error while trying to file : "+e.getMessage(), e);
}
The object created from the content of the text file is then used to create my Source and my Sink and also to do some internal computing in a map function.
How can I correctly load this file in cluster mode ?
Edit :
When I replace the code above with
myObject = new MyClassParser().parse(FILE_CONTENT_AS_STRING);
The code run well. It looks like it is only the loading of the file that cause the error.
Once you build your code into a jar, the file path no longer can be used to access the file. Instead, you need to use something like <YourClassName>.class.getResourceAsStream("/my_file.txt")
to get the InputStream
.