Search code examples
pythonlistdictionaryunique

How to remove duplicate dictionary based on selected keys from a list of dictionaries in Python?


I am new to Python and trying to learn it as much as possible. I am stuck with a silly problem where I want to remove certain dictionary items of a list based on selective key-value pairs. For ex, I have:

l = [{'A':1, 'B':2, 'C':3, 'D':4},
     {'A':5, 'B':6, 'C':7, 'D':8},
     {'A':1, 'B':9, 'C':3, 'D':10}]

And the output I want is removal of dictionaries based on two keys A and C values:

l = [{'A':1, 'B':2, 'C':3, 'D':4},
     {'A':5, 'B':6, 'C':7, 'D':8}]

Solution

  • Using set to remember whether the items are seen.

    >>> A, B, C, D = 'ABCD'
    >>>
    >>> lst = [
    ...     {A:1, B:2, C:3, D:4},
    ...     {A:5, B:6, C:7, D:8},
    ...     {A:1, B:9, C:3, D:10}
    ... ]
    >>> seen = set()
    >>> [x for x in lst if [(x[A], x[C]) not in seen, seen.add((x[A], x[C]))][0]]
    [{'A': 1, 'C': 3, 'B': 2, 'D': 4}, {'A': 5, 'C': 7, 'B': 6, 'D': 8}]