Search code examples
mongodbbashshellcroncron-task

Write a Bash shell script to read mongodb log file and write it's content to another text file

I am trying to read the MongoDB log file located at /var/log/mongodb it's contents are as such:

2019-11-04T05:04:00.390-0800 I COMMAND [conn38649] command loldb.$cmd command: update { update: "SUBSCRIPTION", ordered: true, writeConcern: { w: 1 }, $db: "loldb" } numYields:0 reslen:295 locks:{ Global: { acquireCount: { r: 460, w: 460 } }, Database: { acquireCount: { w: 460 } }, Collection: { acquireCount: { w: 459 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_query 568ms

2019-11-04T05:04:00.396-0800 I COMMAND [conn38657] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824180, policeDepartmentID: 1 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:69998 cursorExhausted:1 numYields:550 nreturned:1 reslen:430 locks:{ Global: { acquireCount: { r: 1102 } }, Database: { acquireCount: { r: 551 } }, Collection: { acquireCount: { r: 551 } } } protocol:op_query 424ms

2019-11-04T05:04:00.402-0800 I COMMAND [conn38735] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824164 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:58142 cursorExhausted:1 numYields:456 nreturned:1 reslen:417 locks:{ Global: { acquireCount: { r: 914 } }, Database: { acquireCount: { r: 457 } }, Collection: { acquireCount: { r: 457 } } } protocol:op_query 374ms

Each blockquote is a single line entry

The contents of file update each second I need to read the file and if the query time protocol:op_query 385ms is more than 300ms I need to save that entire log/line into another text file slow_queries.text.

The file from which I am reading is .log file but the content seem like JSON format (please correct me if I am wrong) preceded by timestamp and command type, is there any efficient way to read data of this format? I am just reading word by word line by line.

Also, what do I do so that the changes made to the .log file are automatically read without running the script every time?

Solution

  • I just tried this on my local machine, maybe needs some work for your usecase. But I added some comments, so maybe this will help you:

    EDIT: I added a check for the timestamp, you would have to configure it to your needs

    #!/bin/bash

# continously read from the file and pipe it into the while loop
tail -F "test.log" | \
while read LINE
do
    # get timestamp from LINE and get time in seconds
    timeinseconds="$(grep -P "^\S*" | date -d - +%s)"
    # get current timestamp before 5 minutes
    timebeforefivemin="$(date -d '-5 minutes' +%s)"
    # only log if timestamp of line is smaller to time before 5 min
    if [[ $(expr $timeinseconds - $timebeforefivemin) -lt 0 ]];
        then
        # get the time of the query from the line
        querytime="$(echo "$LINE" | grep -oP '\d+ms' | grep -oP '\d+')"
        #if the grep was successful and the query time is greater than 300
        if [ $? = 0 ] && [ "$querytime" -gt 300 ]
        then
            # echo the line into the slow_queries file -> change it to the path you want
            echo "$LINE" >> slow_queries.txt
        fi
    fi
done