How to generate unique log files with Slf4j and logback?

Question

I have a spring boot application that uses Slf4j and logback.

This application contains the following class:

@Component
public class CustomFileLogger {
    private FileAppender<ILoggingEvent> mainLogAppender;

    // Start logging to the main log directory
    public void startMainLog(String uniquePath, String uniqueId) {
        mainLogAppender = createFileAppender(uniquePath + "/" + uniqueId + ".log", uniqueId);
        attachAppender(mainLogAppender);
    }

    // Create a file appender
    private FileAppender<ILoggingEvent> createFileAppender(String filePath, String uniqueId) {
        LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();

        FileAppender<ILoggingEvent> fileAppender = new FileAppender<>();
        fileAppender.setContext(context);
        fileAppender.setName(uniqueId);
        fileAppender.setFile(filePath);

        PatternLayoutEncoder encoder = new PatternLayoutEncoder();
        encoder.setContext(context);
        encoder.setPattern("[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%thread] [%level] [%logger{0}.%M:%line] - %msg%n");
        encoder.start();

        fileAppender.setEncoder(encoder);
        fileAppender.start();

        return fileAppender;
    }

    // Attach appender to root logger
    private void attachAppender(FileAppender<ILoggingEvent> appender) {
        Logger rootLogger = (Logger) LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME);
        rootLogger.addAppender(appender);
    }

    // Stop logging and remove the main log appender
    public void stopMainLog() {
        Logger rootLogger = (Logger) LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME);
        if (mainLogAppender != null) {
            rootLogger.detachAppender(mainLogAppender);
            mainLogAppender.stop();
            mainLogAppender = null;
        }
    }
}

CustomFileLogger class is being used by this class:

@Setter
@Slf4j
public abstract class AbstractService {
    @Autowired
    private ApplicationContext applicationContext;

    @Autowired
    protected ServiceA serviceA;

    @Autowired
    protected ServiceB serviceB;

    protected Params params;

    protected CustomFileLogger customFileLogger;

    public void process() {
        try {
            customFileLogger = applicationContext.getBean(CustomFileLogger.class);
            customFileLogger.startMainLog(params.getUniquePath(), params.getUniqueID());

            prepareData();
            generateReport();
            
        } catch (CustomException e) {
            handleExeption(e);
        } finally {
            customFileLogger.stopMainLog();
        }
    }
    // other methods
}

The desired behavior is to have each instance of an AbstractService subclass logging messages to a dedicated file based on a unique ID and path associated with that instance. I have several prototype-scoped subclasses (SubclassA, SubclassB, and SubclassC) that run sequentially, each sharing a unique ID and path for the current run.

For example, if uniqueId = xptoID, I expect all log messages from SubclassA, SubclassB, and SubclassC instances with this unique ID to be appended to a single log file named xptoID.log.

The problem is that, when multiple instances of a subclass (e.g., multiple SubclassA instances) run concurrently, log messages intended for one unique log file sometimes end up in files belonging to other instances. This results in log files that contain messages from different, unrelated executions.

Answer 1

There is no need to create a custom class to handle this kind of scenario. You can use SiftingAppender for this.

I assume that your spring boot application has a file "logback-spring.xml" inside the resources folder.

In this file add something like this:

<appender name="FILE-THREAD"
        class="ch.qos.logback.classic.sift.SiftingAppender">
    <discriminator>
        <key>logKey</key>
        <defaultValue>/path/to/logs/defaultFileName</defaultValue>
    </discriminator>

    <sift>
        <appender name="FILE-${logKey}"
            class="ch.qos.logback.core.rolling.RollingFileAppender">
            <file>${logKey}.log</file>
            <encoder>
                <pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%thread] [%level] [%logger{40}] - %msg%n</pattern>
                <charset>utf8</charset>
            </encoder>

            <filter class="ch.qos.logback.classic.filter.LevelFilter">
                <level>DEBUG</level>
                <onMatch>DENY</onMatch>
                <onMismatch>ACCEPT</onMismatch>
            </filter>

            <rollingPolicy
                class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                <fileNamePattern>${logKey}.%d{yyyy-MM}.log</fileNamePattern>
                <maxHistory>12</maxHistory>
            </rollingPolicy>
        </appender>
    </sift>
</appender>

You will need to create a FileLogger class, eg:

import org.slf4j.MDC;

public class FileLogger {
    protected FileLogger() {
        throw new IllegalStateException("Logger class");
    }
    //use this method to start logging
    public static void setLogKey(String logKey) {
        MDC.put("logKey", logKey);
    }
    //always remove the key after the logging
    public static void removeLogKey() {
        MDC.remove("logKey");
    }
}

Then in your AbstractService:

public abstract class AbstractService {
    // same code
    protected FileLogger fileLogger;

    public void process() {
        try {
            FileLogger.setLogKey(params.getUniquePath() + params.getUniqueID());

            prepareData();
            generateReport();
            
        } catch (CustomException e) {
            handleExeption(e);
        } finally {
            FileLogger.removeLogKey();
        }
    }
    // other methods
}

So, whenever an implementation of AbstractService is logging a message, the content will be written inside a log file composed by the logKey. Other logging messages from your spring application will be added to the defaultFileName.

LevelFilter and TimeBasedRollingPolicy is just an example.