Search code examples
pythonpandascsvfor-in-loop

Add a pause bewteen a for-in loop


I have a script that accesses a website over 100 times and want to add a small delay between each item. Also, if possible, maybe add a countdown or process bar displaying the status or number of items left. The code example is below;

import pandas as pd

urls = ['https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/440/vehicletype/car?format=csv', 
    'https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/441/vehicletype/car?format=csv', 
    'https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/442/vehicletype/car?format=csv', 
    'https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/443/vehicletype/car?format=csv', 
    'https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/445/vehicletype/car?format=csv', 
    'https://vpic.nhtsa.dot.gov/api/vehicles/GetModelsForMakeIdYear/makeId/448/vehicletype/car?format=csv']             


dfs = [pd.read_csv(url) for url in urls]
df = pd.concat(dfs, ignore_index=True)                                                                                      
df.to_csv('foo.csv')                                                                                                        

Solution

  • import time 
    dfs = []
    num = len(urls)
    for i, url in enumerate(urls):
      dfs.append(pd.read_csv(url))
      print 'Process {}, {} left'.format(i+1, num-i-1)
      time.sleep(2)