Search code examples
archivenlogsplunk

Date in NLog file name and limit the number of log files

I'd like to achieve the following behaviour with NLog for rolling files:
1. prevent renaming or moving the file when starting a new file, and
2. limit the total number or size of old log files to avoid capacity issues over time

The first requirement can be achieved e.g. by adding a timestamp like ${shortdate} to the file name. Example: 

logs\trace2017-10-27.log <-- today's log file to write   
logs\trace2017-10-26.log  
logs\trace2017-10-25.log  
logs\trace2017-10-24.log <-- keep only the last 2 files, so delete this one

According to other posts it is however not possible to use date in the file name and archive parameters like maxArchiveFiles together. If I use maxArchiveFiles, I have to keep the log file name constant: 

logs\trace.log <-- today's log file to write   
logs\archive\trace2017-10-26.log  
logs\archive\trace2017-10-25.log  
logs\archive\trace2017-10-24.log <-- keep only the last 2 files, so delete this one

But in this case every day on the first write it moves the yesterday's trace to archive and starts a new file.

The reason I'd like to prevent moving the trace file is because we use Splunk log monitor that is watching the files in the log folder for updates, reads the new lines and feeds to Splunk.

My concern is that if I have an event written at 23:59:59.567, the next event at 00:00:00.002 clears the previous content before the log monitor is able to read it in that fraction of a second.

To be honest I haven't tested this scenario as it would be complicated to set up as my team doesn't own Splunk, etc. - so please correct me if this cannot happen.

Note also I know that it is possible to directly feed Splunk other ways like via network connection, but the current setup for Splunk at our company is reading from log files so it would be easier that way.

Any idea how to solve this with NLog?

Solution

  • When using NLog 4.4 (or older) then you have to go into Halloween mode and make some trickery.

    This example makes hourly log-files in the same folder, and ensure archive cleanup is performed after 840 hours (35 days):

        fileName="${logDirectory}/Log.${date:format=yyyy-MM-dd-HH}.log"
    archiveFileName="${logDirectory}/Log.{#}.log"
    archiveDateFormat="yyyy-MM-dd-HH"
    archiveNumbering="Date"
    archiveEvery="Year"
    maxArchiveFiles="840"
    • archiveFileName - Using {#} allows the archive cleanup to generate proper file wildcard.
    • archiveDateFormat - Must match the ${date:format=} of the fileName (So remember to correct both date-formats, if change is needed)
    • archiveNumbering=Date - Configures the archive cleanup to support parsing of filenames as dates.
    • archiveEvery=Year - Activates the archive cleanup, but also the archive file operation. Because the configured fileName automatically ensures the archive file operation, then we don't want any additional archive operations (Ex. avoiding generating extra empty files at midnight).
    • maxArchiveFiles - How many archive files to keep around.

    With NLog 4.5 (Still in BETA), then it will be a lot easier (As one just have to specify MaxArchiveFiles). See also https://github.com/NLog/NLog/pull/1993