Search code examples
javajacksonapache-flink

Should you import your own version of Jackson into a Flink Project?


Should you import Jackson into a Flink project separately or just use the flink-shaded version of Jackson? And why?


Solution

  • Using the shaded Jackson with older versions of Flink is a bad idea.

    We've seen failures caused by the following class loader leak:

    • user decodes JSON with their user-space classes
    • Jackson caches these classes internally
    • this cache is in Flink's classloader if used from the shaded dependencies
    • user-code class loader cannot be unloaded as long as this class is in the cache (can be quite long)
    • Job restarts or session clusters will amplify this problem

    This was fixed with FLINK-29834 in 1.16.1 and 1.17.0.