Search code examples
unixawksolaris

awk - Find duplicate entries in 2 columns, keep 1 duplicate and unique entries

I need to find a duplicate entry in 2 different columns and keep only one of the duplicate and all unique entries. For me if A123 is in the first column and it show up later in the second column it's a duplicate. I also know for sure that A123 will always be paired to B123 by either being A123,B123 or B123,A123. I only need to keep one and it doesn't matter which one it is.

Ex: My input file would contain:
A123,B123
A234,B234
C123,D123
B123,A123
B234,A234

I'd like the output to be:
A123,B123
A234,B234
C123,D123

The best I can do is to extract the unique entries with : 

awk -F',' 'NR==FNR{x[$1]++;next}; !x[$2]' file1 file1

or get only the duplicates with

awk -F',' 'NR==FNR{x[$1]++;next}; x[$2]' file1 file1


Any help would be greatly appreciated.

Solution

  • This can be shorter!

    First print if the element is not yet present in the array. Then add the first field to the array. Only one run over the inputfile is necessary:

    awk -F, '!x[$2];{x[$1]++}' file1