Search code examples
pythonsortingnumpyunique

How to remove duplicates from a list of tuples but keeping the original order


I want to remove redundant tuples but preserve the order of appearance. I looked at similar questions. This question Find unique rows in numpy.array looked very promising but somehow it did not work for me.

I could use pandas as in this answer (https://stackoverflow.com/a/14089586/566035) but I prefer not using pandas so that the py2exe generated executable will be small.

import numpy as np

data = [('a','z'), ('a','z'), ('a','z'), ('1','z'), ('e','z'), ('c','z')]

#What I want is:
    array([['a', 'z'],
           ['1', 'z'],
           ['e', 'z'],
           ['c', 'z']], 
          dtype='|S1')

#What I have tried:
# (1) numpy.unique, order not preserved
np.unique(data)

    array([['a', 'z'],
           ['c', 'z'],
           ['1', 'z'],
           ['e', 'z']], 
          dtype='|S1')

# (2) python set, order not preserved
set(data)

    set([('1', 'z'), ('a', 'z'), ('c', 'z'), ('e', 'z')])

# (3) answer here : https://stackoverflow.com/a/16973510/566035, order not preserved
a = np.array(data)
b = np.ascontiguousarray(a).view(np.dtype((np.void, a.dtype.itemsize * a.shape[1])))
_, idx = np.unique(b, return_index=True)

a[idx]

    array([['1', 'z'],
           ['a', 'z'],
           ['c', 'z'],
           ['e', 'z']], 
          dtype='|S1')

Solution

  • This isn't great in terms of efficiency, but is very simple readable code and can work for smaller lists:

    sorted(set(data), key=data.index)