Recently I updated my spark version to 3.3. Before update, my log4j.properties works fine, and the log show me only errors. But after update, my log4j.properties don't work anymore.
This is my log4j.properties:
# Set root logger level to desired log level (DEBUG, INFO, WARN, ERROR, or FATAL)
log4j.rootLogger=ERROR, console
# Define console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Set log level for specific packages or classes
# For example, to set the log level to WARN for the org.apache.spark package:
log4j.logger.org.apache.spark=WARN
Aparently log4j.properties don't work with spark 3.3. After performing several tests I realized that spark was setting as default a file called log4j2.default. So to test, I updated my log4j.properties to log4j2.properties and it worked.