Search code examples
scalaintellij-ideaapache-spark

How to set up logging level for Spark application in IntelliJ IDEA?

I'm working on a Scala project in IntelliJ that was created through SBT. The project has Spark as one of its dependencies. I'm still in the development phase so everything is running on my local machine.

How can I change Spark configurations like logging level, for example?

Solution

  • If you are working on the local development with IDE, you can change the log level at run-time by:

    LogManager.getRootLogger.setLevel(Level.ALL)

    Ps: Put that line after the SparkContext/ SQLContext was created in your code.