Search code examples
apache-flink

Checkpoint abort on not executing operator

We have 2 sources.

  1. In memory source from an object
  2. Kafka stream subscribing source

They both connect at some point

Source(from objects) -> Map1 -> CoMap -> Sink
Source(Kafka) -> Map2 -> Map3 -> CoMap -> ...

The Map1 operator is forced to be non parallel

For some reason sometimes we get this kind of log message:

INFO  Checkpoint triggering task Source: Source(from objects) -> Map1 (1/1) is not being executed at the moment. Aborting checkpoint.

We don't really care about checkpoints for the Source(from objects) -> Map1 chain. Can we disable checkpoints there or disable the restoration of them somehow?

Solution

  • The problem was due to some operators finishing and not running. When an operator is not running no checkpoints can happen.

    The solution was to implement a custom SourceFunction and block the thread forever after all the elements were sent