Search code examples
awkcygwingawkgnuwin32

awk keep if line contains " example "

okay so I wish to keep lines containing several keywords,

example of list:

Name:email:username #registered
Name2:email2:username2
Name3:email3:username3 #registered #subscribed #phonever
Name4:email4:username4 #unconfirmed

What I want to do is extract lines if they contain " #registered, #subscribed, #phonever

example of output I want,

Name:email:username #registered
Name3:email3:username3 #registered #subscribed #phonever
Solution

  • With awk (use regex alternation operator, |, on a list of fixed strings):

    awk '/#registered|#subscribed|#phonever/' file

    The part under /.../ is called an awk pattern and for the matching lines it executes the action that follows (as { ... }). But since the default action is: { print $0 } (printing the complete input record/line), there's no need to specify it here.

    Similarly with sed you could say:

    sed -nE '/#registered|#subscribed|#phonever/p' file

    but now we have to specify -n to skip printing by default, and print with the p command only those lines that match the pattern (called sed address). The -E tells sed to used POSIX ERE (extended regex), and we need it here, because the default, POSIX BRE (basic regex) does not define the alternation operator.

    For simple filtering (and printing the lines that match some pattern), grep is also an option (and a very fast option at that):

    grep '#registered\|#subscribed\|#phonever' file

    A bit more general solution (awk with patterns file)

    Solution for larger (and possibly dynamic) lists of patterns could be to keep all patterns in a separate file, for example in patterns:

    #registered
#subscribed
#phonever

    and to use this awk program:

    awk 'NR==FNR { pat[$0]=1 } NR>FNR { for (p in pat) if ($0 ~ p) {print;next} }' patterns file

    which will first load all patterns into pat array, and then try to match any of those patterns on each of the lines in file, printing and advancing on to the next line on the first match found.

    The result is the same:

    Name:email:username #registered
Name3:email3:username3 #registered #subscribed #phonever

    but the script now doesn't change for each new set of patterns. Note however, this caries a performance penalty (as general solutions usually do). For shorter lists of patterns and smaller files, this shouldn't be a problem.

    And a lot faster variant of the above (grep with fixed-string patterns file)

    Building on the approach from above (of keeping a list of fixed-string "patterns" in a file), we can actually use grep -- which provides a specialized option (-f FILE) for obtaining patterns from file, one per line. To further speed-up the matching, we should also use -F/--fixed-strings option.

    So, this:

    grep -Ff patterns file

    will be incredibly fast, handling long lists of fixed-string patterns and huge files with minimal memory overhead.