Search code examples
pythonfor-loopweb-scrapingiterator

enter data and iterate to scrape information on a webpage

I'm trying to scrape this website using python.

https://solanamonkey.business/search

Where I'm running into issues I'd like to iterate through 1-5000 entered in the search field and scraping the data displayed into a .CSV

Any advice on the best method?

Solution

  • Theres a direct url to fetch this:

    import requests
import pandas as pd

rows = []
for i in range(1,5001):
    url = 'https://solanamonkey.business/.netlify/functions/fetchSMB?id=%s' %i
    jsonData = requests.get(url).json()
    
    title = jsonData['nft']['id']
    row = {'id':title}
    for each in jsonData['nft']['attributes']['attributes']:
        row.update({each['trait_type']: each['value']})
    rows.append(row)

    print('ID: ', title)

df = pd.DataFrame(rows)   
df.to_csv('file.csv', index=False)