I want to save my list of dictionaries into a csv file, but I get a UnicodeEncodeError

Question

I have a parsed data that I have scraped but to make this data useful I want it to be saved in a CSV file.

I used this following code to convert the list of dictionaries to a CSV file.

# Example run:
if __name__ == "__main__":
    with httpx.Client(timeout=httpx.Timeout(20.0)) as session:
        posts = list(scrape_user_posts("1142370320", session, max_posts=25, page_size=3))
        print(json.dumps(posts, indent=2, ensure_ascii=False))
        keys = posts[0].keys()
        with open('Lenskart_posts.csv', 'w', newline='') as output_file:
            dict_writer = csv.DictWriter(output_file, keys)
            dict_writer.writeheader()
            dict_writer.writerows(posts)

The error that the compiler throws back is this

Traceback (most recent call last):
File "C:\Users\arjun\AppData\Local\Programs\Python\Python311\Scrapping_test.py", line 108, in <module>
dict_writer.writerows(posts)
File "C:\Users\arjun\AppData\Local\Programs\Python\Python311\Lib\csv.py", line 157, in writerows
return self.writer.writerows(map(self._dict_to_list, rowdicts))
File "C:\Users\arjun\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f31f' in position 878: character maps to <undefined>

How do I fix this error and extract the required CSV file?

Answer 1

To fix the UnicodeEncodeError, I would suggest replacing the following line of code:

with open('Lenskart_posts.csv', 'w', newline='') as output_file:

with this:

with open('Lenskart_posts.csv', 'w', newline='', encoding='utf-8') as output_file:

You can also specify a different encoding if necessary for your situation.

Hope this helps!