Search code examples
awkcut

How to remove some rows based on repetitive values in the second column in linux?


I have a dataset as below. I want to remove rows based on 2 condition: 1: if the values in the second column are repeated less than 5 times all the rows related that column should be removed 2: if the values in the second column are repeated more than 5 times the extra after 5 times repeat should be removed.

Here is the input:

`

7466655 6974128 1 -0.5369 25.79
7466657 6974128 1 1.2966 80.20
7466659 6974128 1 1.5637 25.79
7466660 6974128 1 -0.3545 25.79
7466661 6974128 1 2.4080 25.79
7466663 6974128 1 3.3537 25.79
7466664 6974128 1 0.7214 37.94
7466793 6974080 1 -0.7481 26.28
7466791 6974080 1 -0.7424 26.28
7466790 6974080 1 -0.7224 26.28
8069261 7466657 1 -3.8792 25.95
8069264 7466657 1 7.3225 25.95
8069266 7466657 1 1.4466 25.95
8069365 7466009 1 3.4094 26.26
8069366 7466009 1 6.7698 26.26
8069367 7466009 1 0.0093 26.26
8069370 7466009 1 -4.9916 26.26
8069382 7466009 1 -2.7491 26.26
8069384 7466009 1 -4.0390 26.26
8089183 7466115 1 -3.0163 26.35


output:

7466655 6974128 1 -0.5369 25.79
7466657 6974128 1 1.2966 80.20
7466659 6974128 1 1.5637 25.79
7466660 6974128 1 -0.3545 25.79
7466661 6974128 1 2.4080 25.79
8069365 7466009 1 3.4094 26.26
8069366 7466009 1 6.7698 26.26
8069367 7466009 1 0.0093 26.26
8069370 7466009 1 -4.9916 26.26
8069382 7466009 1 -2.7491 26.26

Any suggestion is appreciated.


Solution

  • EDIT: Adding more generic solution with a variable where one could set the limit of occurrences condition check. Advantage will be we need not to change values everywhere, simply change value in variable occur.

    awk -v occur="5" 'FNR==NR{a[$2]++;next} a[$2]<occur{next} a[$2]>=occur{if(++b[$2]<=occur){print}}'  Input_file  Input_file
    


    Could you please try following.

    awk 'FNR==NR{a[$2]++;next} a[$2]<5{next} a[$2]>=5{if(++b[$2]<=5){print}}'  Input_file  Input_file
    

    Output will be as follows.

    7466655 6974128 1 -0.5369 25.79
    7466657 6974128 1 1.2966 80.20
    7466659 6974128 1 1.5637 25.79
    7466660 6974128 1 -0.3545 25.79
    7466661 6974128 1 2.4080 25.79
    8069365 7466009 1 3.4094 26.26
    8069366 7466009 1 6.7698 26.26
    8069367 7466009 1 0.0093 26.26
    8069370 7466009 1 -4.9916 26.26
    8069382 7466009 1 -2.7491 26.26