Search code examples
apache-flink

Setting a class member value in the constructor of a KeyedProcessFunction class does not work

I have a class which extends KeyedProcessFunction in Apache Flink.

The class has a transient member, for example a SomeType ok

In the constructor of the said class, the class member ok is population to a SomeType value.

However when the ok is read by the processElement function, it complains that the ok is null.

SomeType is not serialisable and hence it is transient. Is that the reason why this is not working as expected? If so, how should I pass some state into the class which would be read by every element irrespective of the key.

Solution

  • All data going from JobManager's to TaskManager's needs to be serializable due to Flink distributed nature. This data is passed from JM to TM via network and that's the reason why they need to be serializable.

    A way to work around this is to pass the primitive/serializeable pieces via the constructor and use the method RichFunction.open to initialize what is not serializable.