Search code examples
javaapache-sparklog4j2mdc

Spark application and logging MDC (Mapped Diagnostic Context)

I'm searching for a way, to forward my driver's MDC to master and workers, such that I can identify messages stemming from the same operation? E.g. I have a application dependent query/job-id which I want to have attached to structured log messages in my spark workers.

  • Impossible?
  • Undocumented?
  • Easy, but I was to dumb to find?

Regards.

Solution

  • If I understand the PR correctly (I am new to Spark) then since version 3.1 it should be now possible to add custom entries to MDC via spark.sparkContext.setLocalProperty("mdc." + name, "value").