Search code examples
windowsdockerapache-flinkcheckpoint

Path for Flink state.checkpoints.dir in docker-compose in Windows 10 environment

I have Windows 10 OS, docker-compose and want to work with Apache Flink tutorial Playground, docker-compose starting correctly starting docker-compose but after several minutes of work, Apache Flink has to create checkpoints, but there is some problem with access to the file system.

Exception:

org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 104. Failure reason: Failure to finalize checkpoint.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1216) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
…..

Caused by: org.apache.flink.util.SerializedThrowable: Mkdirs failed to create file:/tmp/flink-checkpoints-directory/d73c2f87b0d7ea6748a1913ee4b50afe/chk-104
    at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:262) ~[flink-dist_2.11-1.12.1.jar:1.12.1]

Could you please help me with the correct path and docker access?

state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints-directory
state.savepoints.dir: file:///tmp/flink-savepoints-directory

Also I tried use full Windows path but got the same error.

enter image description here

Solution

  • Are you using Windows Docker containers or Linux Docker containers ?

    Right-click on the Docker Desktop icon to see your current configuration.

    • Switch to Windows containers

    OR

    • Switch to Linux containers

    Docker for Windows

    You have to configure Flink paths according to you target Docker container type.

    Note: You cannot use Windows and Linux containers at the same time.