Search code examples
awkgrepmatchcarriage-return

remove all lines in a file containing a string from another file


I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:

grep -vFf to_delete.csv inputfile.csv > output.csv

Here are sample lines from my input file (inputfile.csv):

Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.

My file "to_delete.csv" looks like this for example:

Aqu
Aca

So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:

Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.

Solution

  • EDIT: Since OP had carriage characters in his files so adding solution for that too now.

    cat -v Input_file     ##To check if carriage returns are there or not.
    tr -d '\r' < Input_file > temp_file  &&  mv temp_file Input_file
    

    Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.

    awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1'  to_delete.csv  Input_file  > temp_file  && mv temp_file  Input_file
    

    Explanation: Adding explanation for above code too now.

    awk -F, '                          ##Setting field separator as comma here.
    FNR==NR{                           ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
      a[$0]                            ##Creating an array named a whose index is $0.
      next                             ##next will skip all further statements from here.
    }
    {
      for(i=1;i<=NF;i++){              ##Starting a for loop from value i=1 to till value of NF.
         if($i in a){                  ##checking if $i is present in array a if yes then go into this condition block.
           next                        ##next will skip all further statements(since we DO NOt want to print any matching contents)
         }                             ##Closing if block now.
      }                                ##Closing for block here.
    }                                  ##Closing block which should be executed for 2nd Input_file here.
    1                                  ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
    '  to_delete.csv  Input_file       ##Mentioning Input_file names here now.