Search code examples
javahadoophdfsflume

hadoop flume log4j configuration

If you run a hadoop flume node, as default it generates logs under /var/log/flume using log4j. The files will look like 

/var/log/flume/flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log

According to the flume user guide here, the only way to change the flume log configuration is via flume-daemon.sh which runs flume node using the Flume Environment Variables like:

export FLUME_LOGFILE=flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log
export FLUME_ROOT_LOGGER="INFO,DRFA"
export ZOOKEEPER_ROOT_LOGGER="INFO,zookeeper"
export WATCHDOG_ROOT_LOGGER="INFO,watchdog"

The questions are:

  1. if I want to change the log level from INFO to DEBUG, this is the only place to do it?
  2. Is there a configuration somewhere I can do this?
  3. what about I want to set some packages' log level to DEBUG while others stay INFO?
Solution

  • Noticed that under /etc/flume/conf.empty, there is a log4j.properties. Copied it to /etc/flume/conf, restart the flume node service, the log4j.properties file starts taking effect.

    The overriding order is like flume-env.sh->flume-daemon.sh->log4j.properties.

    eg. if you set up your flume_root_logger to DEBUG in flue-daemon.sh, it will override whatever setting you have for root_logger in your log4j.properties.