Search code examples
pythonjsonbeautifulsoupweb-crawlerscraper

How do i save my Python crawler output to a JSON file?


I recently started with coding and learning Python, and I am currently working on a webcrawler. So it is currently just printing out the search results. What i want is that it saves the data into a JSON file.

import requests
import json
from bs4 import BeautifulSoup

url= "http://www.alternate.nl/html/product/listing.html?navId=11622&tk=7&lk=9419"
r = requests.get(url) 
soup = BeautifulSoup(r.content)

g_data = soup.find_all("div", {"class": "listRow"}) 
for item in g_data: 
try: 
    print item.find_all("span", {"class": "name"})[0].text#1
    print item.find_all("span", {"class": "additional"})[0].text#2
    print item.find_all("span", {"class": "info"})[0].text#3
    print item.find_all("span", {"class": "info"})[1].text#4
    print item.find_all("span", {"class": "info"})[2].text#5
    print item.find_all("span", {"class": "price right right10"})[0].text#6
except: 
    pass     

This is what i want it returns:

{"product1":[{"1":"itemfindallresults1"},{"2":"itemfindallresults2"}]} etc

So how can i do that? Thanks in advance.


Solution

  • A simple JSON usage would be:

    import json
    # open the file "filename" in write ("w") mode
    file = open("filename", "w")
    # just an example dictionary to be dumped into "filename"
    output = {"stuff": [1, 2, 3]}
    # dumps "output" encoded in the JSON format into "filename"
    json.dump(output, file)
    file.close()
    

    Hope this helps.