Search code examples
javaperformancelog4j

log4j performance on huge treatment batch (10-12h)

I would know if someone has observed if a lot of log writes in logfile (300k lines if 4 hours of treatment) could penalize batch performance.

P1: The batch writes a lot of info in the logfile and I'm in doubt if we delete or comment all this log writes in source code the batch performance could be increased and gain 15 min or more in time execution.

We could have a million or more lines in a full batch execution (8-12 hours).

P2: Or database check and log writes could be done in parallel ? But i thought our source code doesn't do that.

Solution

  • Well, yes. Too much logging does affect performance. But the only way to know how much it affects performance would be to measure it.

    P1: The batch writes a lot of info in the logfile and I'm in doubt if we delete or comment all this log writes in source code the batch performance could be increased and gain 15 min or more in time execution.

    Nobody can tell you how much time you would gain. (I'd be surprised if you gained as much as that, but I could be wrong. Measure it!!)

    P2: Or database check and log writes could be done in parallel ? But i thought our source code doesn't do that.

    It is probably a bad idea to explicitly code parallel logging into your application, since it will make your code a lot more complicated. And there is a better way to get some parallelism: try using an asynchronous appender.

    There are a number of things that you can do to tune logging performance without going to the extreme of ripping it all out. These include:

    • Switch to a different logging library. For example, log4j 2.x should be more efficient than log4j 1.2.
    • Don't log too much.
    • Log at an appropriate level, and adjust the log level depending on the circumstances.
    • Make sure that you are creating the log messages efficiently. For example, avoid generating complicated message strings that won't be logged due to the logging level. (In log4j 2.x, use the Logger methods that take format strings.)
    • Avoid expensive features in your log format / formatter. For instance logging the class / method is relatively expensive.
    • Try using an asynchronous log appender.

    For some background on logging performance, take a look at the log4j2 Performance page.