How to set log filename in flume

Question

I am using Apache flume for log collection. This is my config file

httpagent.sources = http-source
httpagent.sinks = local-file-sink
httpagent.channels = ch3

#Define source properties 

httpagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 8082


# Local File Sink

httpagent.sinks.local-file-sink.type = file_roll
httpagent.sinks.local-file-sink.channel = ch3
httpagent.sinks.local-file-sink.sink.directory = /home/avinash/log_dir
httpagent.sinks.local-file-sink.sink.rollInterval = 21600

# Channels

httpagent.channels.ch3.type = memory
httpagent.channels.ch3.capacity = 1000

My application is working fine.My problem is that in the log_dir the files are using some random number (I guess its timestamp) timestamp as by default.

How to give a proper filename suffix for logfiles ?

Answer 1

Having a look on the documentation it seems there is no parameter for configuring the name of the files that are going to be created. I've gone to the sources looking for some hidden parameter, but there is no one :)

Going into the details of the implementation, it seems the name of the file is managed by the PathManager class:

private PathManager pathController;
...
@Override
public Status process() throws EventDeliveryException {
    ...
    if (outputStream == null) {
        File currentFile = pathController.getCurrentFile();
        logger.debug("Opening output stream for file {}", currentFile);
        try {
            outputStream = new BufferedOutputStream(new FileOutputStream(currentFile));
    ...
}

Which, as you already noticed, is based on the current timestamp (showing the constructor and the next file getter):

public PathManager() {
    seriesTimestamp = System.currentTimeMillis();
    fileIndex = new AtomicInteger();
}

public File nextFile() {
    currentFile = new File(baseDirectory, seriesTimestamp + "-" + fileIndex.incrementAndGet());
    return currentFile;
}

So, I think the only possibility you have is to extend the File Roll sink and override the process() method in order to use a custom path controller.