Logging in Databricks/Spark

Question

I am quite new to spark/databricks and I know that there is a logging in Azure. My idea is to have a log like a print, directly in the databricks notebook. Somebody said to me to use the native spark lib, but I could´t find anything anywhere about that.

Thank you and sorry :)

EDIT: I tried doing this log in a notebook in databricks with Python:

import logging
logger = logging.getLogger(__name__)
logger.warning("This is a warning message") #it prints to the notebook
logger.error("This is an error message") #this also prints to the notebook, but the following don't print to the notebook:
logger.debug("About to do something")
logger.info("This is an informational message")

I wanted to know something like this, but this is from Python, and I wanted to know if there is a logging behaving like a print lib in Spark, something like this:

import loging_from_Spark_lib
logger("The process 1 is starting...")

and it prints the message: "The process 1 is starting..." to the notebook.

I don't know if there is a native library from Spark, my boss asked me that and I could't find it. Plus if there isn't is there a better way of doing this in databricks? Once again Thanks :)

Answer 1

Even, I got same in my environment.

This is because of Log Levels.

According to the levels in this documentation the messages will be logged. Default it will be Warning, below levels are not logged. So, set the log level which ever you want and log.

Try below code block.

import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.debug("About to do something")
logger.info("This is an informational message")