Search code examples
bashshelllogging

Writing a shell script to extract the logs within the specified timestamp

Below is the script. When I try to execute it, it says there is an error parsing the date.

[17:02:04:308][01-03-2024] <-- This is the structure of the timestamp in my log file. The error in the terminal:

./extract_logs.sh serverout.txt "[17:02:04:308][01-03-2024]" "[17:02:04:325][01-03-2024]"
date: invalid date ‘ 17:02:04:308  01-03-2024 ’
date: invalid date ‘ 17:02:04:325  01-03-2024 ’
date: invalid date ‘ 17:02:04:227  01-03-2024 ’
./extract_logs.sh: line 39: [: : integer expression expected
date: invalid date ‘ 17:02:04:303  01-03-2024 ’
./extract_logs.sh: line 39: [: : integer expression expected
date: invalid date ‘ 17:02:04:304  01-03-2024 ’
./extract_logs.sh: line 39: [: : integer expression expected
date: invalid date ‘ 17:02:04:308  01-03-2024 ’
./extract_logs.sh: line 39: [: : integer expression expected

The code:

#!/bin/bash

# Function to convert timestamp to seconds since epoch
convert_to_seconds() {
    local timestamp=$1
    date -d "$timestamp" +%s
}

# Check if correct number of arguments are passed
if [ "$#" -ne 3 ]; then
    echo "Usage: $0 <log_file_name> <start_timestamp> <end_timestamp>"
    echo "Timestamps should be in the format '[HH:MM:SS:SSS][DD-MM-YYYY]'"
    exit 1
fi

log_file=$1
start_timestamp=$2
end_timestamp=$3

# Convert start and end timestamps to seconds since epoch
start_seconds=$(convert_to_seconds "$start_timestamp")
end_seconds=$(convert_to_seconds "$end_timestamp")

# Check if log file exists
if [ ! -f "$log_file" ]; then
    echo "File not found: $log_file"
    exit 1
fi

# Read the log file line by line
while IFS= read -r line; do
    # Extract the timestamp from the log line
    if [[ $line =~ (\[[0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3})\]\[([0-9]{2}-[0-9]{2}-[0-9]{4})\] ]]; then
        log_time="[${BASH_REMATCH[1]}][${BASH_REMATCH[2]}]"
        log_seconds=$(convert_to_seconds "$log_time")

        # Check if log time is within the specified range
        if [ "$log_seconds" -ge "$start_seconds" ] && [ "$log_seconds" -le "$end_seconds" ]; then
            echo "$line"
        fi
    fi
done < "$log_file"
Solution

  • The simple explanation is that date stamps in your crazy format are not acceptable to date.

    A workaround is to convert the dates to something sane; but since these are just numbers, you don't really need to convert the value to seconds -- just do a simple string comparison after reordering the fields.

    #!/bin/bash
 
# Check if correct number of arguments are passed
if [ "$#" -ne 3 ]; then
    echo "Usage: $0 <log_file_name> <start_timestamp> <end_timestamp>"
    echo "Timestamps should be in the format '[HH:MM:SS:SSS][DD-MM-YYYY]'"
    exit 1
fi
 
log_file=$1
start_timestamp=$2
end_timestamp=$3
 
# Check if log file exists
if [ ! -f "$log_file" ]; then
    echo "File not found: $log_file"
    exit 1
fi
 
awk -v start="$start_timestamp" -v end="$end_timestamp" '
  function parsedate(date) {
        split(date, a, /[]:[-]+/)
        return a[8] "-" a[7] "-" a[6] "T" a[2] ":" a[3] ":" a[4] "." a[5]
  }
  BEGIN {
    st = parsedate(start)
    et = parsedate(end)
  }
  ((p = parsedate($0)) >= st) && (p <= et)' "$1"

    I refactored the main logic to Awk as that is significantly more succinct than the equivalent Bash code.

    Demo: https://ideone.com/GOgdoP

    Tangentially, perhaps note also that there are multiple versions of date. You are apparently on Linux, which uses GNU date (which supports e.g. the -d option for specifying a date). On e.g. BSD / MacOS you have different options (including an option to specify how exactly to convert a string in some wacky format to a date, but with many other oddities).