How to make fluentd send logs to Loki from the last dynamically created folder

Question

At the moment I collect all multiline logs and visualize them on the Grafana dashboard. And now I need to get logs from the last dynamically created folder and visualize them on the separate Grafana dashboard. The folders are created dynamically each time when the application is started.

logs path is the next:

/var/log/2021-01-13/12_57_12_201/
/var/log/2021-01-13/12_54_48_123/
/var/log/2021-01-14/11_15_23_849/11_15_23_849_A.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_B.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_C.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_D.log

I use the docker-compose file to start grafana:7.3.3, fluent-plugin-loki:latest and loki:2.1.0, and fluent.conf file to collect all logs and send them to Loki. Is it possible to collect logs from the last dynamically created folder? And how is it possible to do?

Here is my fluent.conf file:

<source>
   @type tail
   @log_level debug
   path /var/log/%Y-%m-%d/*/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag varlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   follow_inodes true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match varlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd"}
  <label>
    filename
  </label>
</match>

After I change my configuration with using fluentd exec input plugin I receive next information in fluentd log: fluent/log.rb:302:debug: Executing command title=:exec_input spawn=[{}, "sh /var/log/folderParser.sh"] mode=[:read] stderr=:discard It appeare every time when bash script should be started.

My current fluent.config file with exec input plugin has the next configuration:

<source>
   @type exec
   command sh /var/log/folderParser.sh
   tag newlog.*
   run_interval 10s
   <parse>
      @type none
   </parse>
</source>

<source>
   @type tail
   @log_level debug
   path /var/log/lastLogs/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag newlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match newlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd", "datacenter":"lastlogs"}
  <label>
    filename
  </label>
</match>

Also here is the script that should be performed by exec plugging (its path /var/log/). All logs are here path /var/log/%Y-%m-%d/time_when_app_started/*.log

# Remove all existing files from the previous run
rm -r lastLogs

# Move to the needed folder - var
cd ../

# Find last created folder in log directory
date_folder=`ls -ltr ./log | grep '^d' | tail -1| awk '{print $NF}'`
cd log/
# Find last created folder in last date directory
last_created_folder=`ls -ltr ./$date_folder | grep '^d' | tail -1| awk '{print $NF}'`

mkdir lastLogs
#sleep 1m
cp ./$date_folder/$last_created_folder/* ./lastLogs

The latest bash script

pwd
cd /var/
pwd

# Find last created folder in log directory
date_folder=`ls -ltr ./log | grep '^d' | tail -1| awk '{print $NF}'`
echo $date_folder

# Change directory to log dir to be able to find the last created folder with the logs
cd log/
pwd

# Find last created folder in last date directory
last_created_folder=`ls -ltr ./$date_folder | grep '^d' | tail -1| awk '{print $NF}'`
echo $last_created_folder
# Find last created folder within time directories
path=$date_folder/$last_created_folder
echo $path

# Check if /lastLogs/logs directory contains last logs
if diff ./$date_folder/$last_created_folder/ ./lastLogs/logs/
then
    echo "Files are the same"
#    cp -u ./$date_folder/$last_created_folder/* ./lastLogs/logs

else
    echo "Files or files scope are different"
    rm -r ./lastLogs/logs/*
    cp -u ./$date_folder/$last_created_folder/* ./lastLogs/logs
    echo "Files are reload"
fi

Here is my fluent.conf file

<source>
    @type exec
    command sh /var/log/folderParser.sh
    tag exec.*
    run_interval 1s
    <parse>
       @type none
    </parse>
</source>

<match exec.*>
  @type stdout
</match>

<source>
   @type tail
   @log_level debug
   path /var/log/lastLogs/logs/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag newlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match newlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd", "datacenter":"lastlogs"}
  <label>
    filename
  </label>
</match>

Answer 1

One possible solution could be to:

• use the exec input plugin with a shell script.
• The shell script will evaluate and move the latest files to a predefined path/location.
• You can tail this predefined path for new files.
• Also, route the output of exec to null.