Search code examples
bashawkmergerecord

using awk to merge multiple lines into single line records while removing some of the data


I have this text file;

>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
   Severity: Medium   Confidence: Low
   Location: ./batch/action_reward.py:250
   More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html
>> Issue: [B303:blacklist] Use of insecure MD2, MD4, MD5, or SHA1 hash function.
   Severity: Medium   Confidence: High
   Location: ./batch/local_runs/get_oapi_stores.py:33
   More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b303-md5
>> Issue: [B605:start_process_with_a_shell] Starting a process with a shell, possible injection detected, security issue.
   Severity: High   Confidence: High
   Location: ./batch/local_runs/get_oapi_stores.py:212
   More Info: https://bandit.readthedocs.io/en/latest/plugins/b605_start_process_with_a_shell.html

and need to convert into single line records, like this;

B608 ./batch/action_reward.py:253
B303 ./batch/local_runs/get_oapi_stores.py:33
B605 ./batch/local_runs/get_oapi_stores.py:212

and so far, have started using awk with record and field separators

that I execute with "awk -f sort.awk filename"

BEGIN { RS = ">>" ; FS = "\n" }

{
      print $1" "$3
}

which is close but not complete ...

^I$
^I$
 Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.^I   Location: ./batch/action_reward.py:253$
^I$
 Issue: [B303:blacklist] Use of insecure MD2, MD4, MD5, or SHA1 hash function.^I   Location: ./batch/local_runs/get_oapi_stores.py:33$
^I$
 Issue: [B605:start_process_with_a_shell] Starting a process with a shell, possible injection detected, security issue.^I   Location: ./batch/local_runs/get_oapi_stores.py:212$

How might I strip the headers / remaining text from the line(s) and remove the extra blank lines it creates?

is there a way to do this with substr or equivalent?

answers using awk, please


Solution

  • One solution could be:

    awk '
     $1==">>"{
       sub(/^\[/, "", $3) # remove first `[`
       sub(/:.*/, "", $3) # remove everything after `:`
       str=$3             # save $3 in variable `str`
     }
     $1=="Location:"{
       print str,$2       # print `str` and $2
     }
    ' file