Search code examples
amazon-s3apache-flinkamazon-emr

Checkpoint s3p flink on EMR


I have problem with checkpoint by s3p in the flink of EMR.

When creating the EMR cluster, I have a tick in Presto and added jar file as instructed at https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html.

But when checking point by s3p in flink, it still reports

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3p'. The scheme is directly supported by Flink through the following plugin: flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.

Can you help me checkpoint s3p on the flink of EMR? Thanks.


Solution

  • Presto in EMR has nothing to do with the flink-s3-fs-presto plugin in Flink. You can leave it unticked in the future (doesn't hurt either except blowing things up).

    The most likely reason is that you forgot to create a subfolder in the plugins folder. Could you give me an ls of your Flink distribution?