I have a list of lists, where each list contains a dictionary and integer. Sometimes duplicate lists occur, and I wish to remove these from the parent list directly. Currently, I am creating a new list and iterating over the old list to ensure only unique values are appended, but I feel this is bad practice. Can this be rewritten to a one-liner with list comprehension, or can the original list be filtered directly instead, for performance enhancement?
TRIAL=[[{'http': '46.101.160.223:80', 'https': '46.101.160.223:80'}, 0],
[{'http': '66.70.178.214:9300', 'https': '66.70.178.214:9300'}, 0],
[{'http': '130.61.100.135:80', 'https': '130.61.100.135:80'}, 0],
[{'http': '157.245.27.9:3128', 'https': '157.245.27.9:3128'}, 0],
[{'http': '185.246.84.7:8080', 'https': '185.246.84.7:8080'}, 0],
[{'http': '185.246.84.7:8080', 'https': '185.246.84.7:8080'}, 0],
[{'http': '130.61.100.135:80', 'https': '130.61.100.135:80'}, 1]]
#We have some duplicates which we want to filter out if there with function
temporary_list=[]
for i in TRIAL:
if i[0] not in [item[0] for item in temporary_list]:
temporary_list.append(i)
temporary_list (desired outcome)
[[{'http': '46.101.160.223:80', 'https': '46.101.160.223:80'}, 0],
[{'http': '66.70.178.214:9300', 'https': '66.70.178.214:9300'}, 0],
[{'http': '130.61.100.135:80', 'https': '130.61.100.135:80'}, 0],
[{'http': '157.245.27.9:3128', 'https': '157.245.27.9:3128'}, 0],
[{'http': '185.246.84.7:8080', 'https': '185.246.84.7:8080'}, 0]]
If you can use pandas
:
import pandas as pd
df = pd.DataFrame(TRIAL, columns=['d', 'i'])
out = df.groupby(df['d'].apply(dict.values).apply(tuple)).last().values.tolist()
print(out)
[[{'http': '130.61.100.135:80', 'https': '130.61.100.135:80'}, 1],
[{'http': '157.245.27.9:3128', 'https': '157.245.27.9:3128'}, 0],
[{'http': '185.246.84.7:8080', 'https': '185.246.84.7:8080'}, 0],
[{'http': '46.101.160.223:80', 'https': '46.101.160.223:80'}, 0],
[{'http': '66.70.178.214:9300', 'https': '66.70.178.214:9300'}, 0]]