Search code examples
apache-sparklogginglog4j

Spark submit: these logs are generated without level

I'm running Spark on Kubernetes, the logs of the spark app are sent to Datadog, are logs are okay and sent correctly except for the first 4 lines directly after spark-submit:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.2.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Since these lines don't have a level attribute attached to them, they are interpreted by Datadog as an Error, which is a bit annoying:

The rest of the logs are sent by Log4j with the level attached to the massage, something like this:

{
  endOfBatch: false
  instant: {
    epochMillisecond: 1686055630113.133
    epochSecond: 1686055630
    nanoOfSecond: 113133000
  }
  level: INFO
  loggerFqcn: org.apache.logging.slf4j.Log4jLogger
  loggerName: com.package.Main
  thread: main
  threadId: 1
  threadPriority: 5
}

So, how to declare the 4 first lines with Info level in the logs.

Solution

  • This warning is because you are using Spark 3.x with Java 11, 2 possible "easy" solutions:

    1. Downgrade Java version from 11 to 8.

    2. Upgrade Spark version to 3.3+