I need to get the last X records of the modsec_audit.log file in bash and write it out to a new file. I know this is possible with tail and sed, but this possibly cuts out at a certain line and causes half of a message to be left in the beginning.
Is there a cleaner way to do this? I just need to get the X last events/items from the log file.
For those not familiar with modsecurity log files, here is an example of the modsec_audit.log file:
--38321e0f-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at 10.211.55.6 Port 80</address>
</body></html>
--38321e0f-H--
Message: Access denied with code 503 (phase 2). Pattern match "block_hack" at ARGS:block_hack. [file "/usr/share/modsecurity-crs/activated_rules/block_hack.conf"] [line "2"] [id "11212"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1443442449908332 455 (- - -)
Stopwatch2: 1443442449908332 455; combined=23, p1=6, p2=15, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/).
Server: Apache/2.4.7 (Ubuntu)
Engine-Mode: "ENABLED"
--38321e0f-Z--
some hacking with awk and tac
start with generous tail, reverse, print top 3 records, reverse
tail -100 log2 | tac | awk -vRS= -vORS="\n\n" 'NR<4' | tac
will print the last 3 records from the original log.
Explanation:
Since the log file can be very large you may want to limit the operation on the last few lines. If your records are 10 lines long, tail a corresponding section that will include at least n*10 lines. As most other tools awk start from the beginning of the file, that's why we reverse the lines with tac print n records and reverse back to the original format.
RS= sets awk to paragraph mode (records separated one or more empty lines), similarly ORS="\n\n" for printing records separated by an empty line. NR<4 condition will be true for records 1,2,3 and will be printed (as the default action of awk). -v is the optional tag to define variables.