Search code examples
linuxshellsortingawkunique

How to unique by column large file via sort command?


i have large list (6 million lines) and want to sort unique by column #2, how to do it by sort command? In example

CONT,000-00-0000,GRAM
BEVE,507-66-6876,IGHT
MICH,000-00-0000,EVINS
CONT,111-11-1111,GRAM

trasnfer to

CONT,000-00-0000,GRAM
BEVE,507-66-6876,IGHT
CONT,111-11-1111,GRAM

Solution

  • $ awk -F, '!array[$2]++' input_file
    CONT,000-00-0000,GRAM
    BEVE,507-66-6876,IGHT
    CONT,111-11-1111,GRAM