Search code examples
pythontraceback

Trouble scanning list for duplicates


Hey so i want to scan this text file of emails and if two of the same emails pop up i want it to be printed if only 1 email is on the list i dont want it to be printed.

It worked for a different text file but now its saying traceback error???

#note make sure found.txt and list.txt are in the 'include' for pycharmfrom collect ions import Counter

print("Welcome DADDY")

with open('myheritage-1-million.txt') as f:
    c=Counter(c.strip().lower() for c in f if c.strip()) #for case-insensitive search
    for line in c:
        if c[line] > 1:
            print(line)

ERROR:

rs/dcaputo/PycharmProjects/searchtoolforrhys/venv/include/search.py
Welcome DADDY
Traceback (most recent call last):
  File "/Users/dcaputo/PycharmProjects/searchtoolforrhys/venv/include/search.py", line 5, in <module>
    c = Counter(c.strip().lower() for c in f if c.strip()) #for case-insensitive search
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 566, in __init__
    self.update(*args, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 653, in update
    _count_elements(self, iterable)
  File "/Users/dcaputo/PycharmProjects/searchtoolforrhys/venv/include/search.py", line 5, in <genexpr>
    c = Counter(c.strip().lower() for c in f if c.strip()) #for case-insensitive search
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc5 in position 2668: invalid continuation byte

Process finished with exit code 1

a list of all emails that are shown up 2 times in that whole text file


Solution

  • The key is the error message at the end:

    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc5 in position 2668: invalid continuation byte

    This error can occur when trying to read a non-text file as text. Your file could be corrupted somehow and has some data (at position 2668) in it that can't be read as text.