Search code examples
pythonpython-3.xurllibpymysql

CSV Returning as Bytes not String Python


The error I am recieving is this:

_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

Where in my code would I open the file in text mode?

import csv
import urllib.request
import pymysql.cursors
connection = pymysql.connect(host='localhost',
                         user='',
                         password='',
                         db='mydb',
                         charset='utf8',
                         cursorclass=pymysql.cursors.DictCursor)
try:
    url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?
    collectionName=Health%20Care&format=csv'
    response =  urllib.request.urlopen(url)
    csv_data = csv.reader(response)
    for row in csv_data:

    cursor.execute('INSERT INTO Financials (names, \
          classes, mark )' \
          'VALUES("%s", "%s", "%s")', 
          row)
finally:
    connection.close()

Solution

  • response =  urllib.request.urlopen(url)
    

    here response returns an iterator on a bytes object,.

    If you know that the csv file is just plain text, you could just insert a generator comprehension to decode the lines:

    csv_data = csv.reader(line.decode() for line in response)
    

    or map (don't you love python 3 map when you can use it without lambda?)

    csv_data = csv.reader(map(bytes.decode,response))
    

    self-contained example:

    import urllib.request,csv
    
    url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?collectionName=Health%20Care&format=csv'
    response =  urllib.request.urlopen(url)
    for row in csv.reader(line.decode() for line in response):
        print(row)
    

    now you're feeding csv.reader with a list of strings, which will work.

    sample output:

    ['CAPR', 'Capricor Therapeutics Inc.', 'NASDAQ Capital Market', 'Healthcare', 'close', '0.9011', '1539955800519', '0.875', '1539979200341', '0.9011', '0.8702', '0.875', 'Close', 'October 19, 2018', '1539979200341', '96625', '', '', '', '0.875', '1539979200341', '0.9011', '0.0261', '0.02983', '1539954158310', '0.9011', '-0.0261', '-0.02896', '', '', '106532', '', '', '', '', '26905263', '-1.72', '3.19', '0.851', '-0.4716351592356688']
    ['AVDL', 'Avadel Pharmaceuticals plc', 'NASDAQ Global Market', 'Healthcare', 'close', '4.24', '1539955800643', '4.1', '1539979200312', '4.24', '4', '4.1', 'Close', 'October 19, 2018', '1539979200312', '78386', '', '', '', '4.1', '1539979200312', '4.2', '0.1', '0.02439', '1539892803066', '4.2', '-0.1', '-0.02381', '', '', '114387', '', '', '', '', '150734712', '-3.8', '11.93', '3.98', '-0.5579009090909092']