we are new to Python and we're currently trying to download from Google image using a specific keyword some images in order to make a dataset. We found a good tutorial here, but we couldn't manage to make the code work. We're using Python 3.6 and Google Chrome 68.0.3440.84 on Windows 10. The code is:
import os
import urllib.request as ulib
from bs4 import BeautifulSoup as Soup
import json
url_a = 'https://www.google.com/search?ei=1m7NWePfFYaGmQG51q7IBg&hl=en&q={}'
url_b = '\&tbm=isch&ved=0ahUKEwjjovnD7sjWAhUGQyYKHTmrC2kQuT0I7gEoAQ&start={}'
url_c = '\&yv=2&vet=10ahUKEwjjovnD7sjWAhUGQyYKHTmrC2kQuT0I7gEoAQ.1m7NWePfFYaGmQG51q7IBg'
url_d = '\.i&ijn=1&asearch=ichunk&async=_id:rg_s,_pms:s'
url_base = ''.join((url_a, url_b, url_c, url_d))
headers = {'User-Agent': 'Chrome/41.0.2228.0 Safari/537.36'}
def get_links(search_name):
search_name = search_name.replace(' ', '+')
url = url_base.format(search_name, 0)
request = ulib.Request(url, None, headers)
json_string = ulib.urlopen(request).read()
page = json.loads(json_string)
new_soup = Soup(page[1][1], 'lxml')
images = new_soup.find_all('img')
links = [image['src'] for image in images]
return links
def save_images(links, search_name):
directory = search_name.replace(' ', '_')
if not os.path.isdir(directory):
os.mkdir(directory)
for i, link in enumerate(links):
savepath = os.path.join(directory, '{:06}.png'.format(i))
ulib.urlretrieve(link, savepath)
if __name__ == '__main__':
search_name = 'referee'
links = get_links(search_name)
save_images(links, search_name)
We already tried to change the user agent but nothing changed. Then, we tried to remove the url_d string in the url_base, which changed the error in:
in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
If someone has any suggestions or ideas, please let us know. Thank you in advance
You are not getting json data back. Change get_links to this:
def get_links(search_name):
search_name = search_name.replace(' ', '+')
url = url_base.format(search_name, 0)
request = ulib.Request(url, None, headers)
data = ulib.urlopen(request).read()
new_soup = Soup(data, 'lxml')
images = new_soup.find_all('img')
links = [image['src'] for image in images]
return links
After running your script with the update:
(661) $ ls referee/
000000.png 000002.png 000004.png 000006.png 000008.png 000010.png 000012.png 000014.png 000016.png 000018.png
000001.png 000003.png 000005.png 000007.png 000009.png 000011.png 000013.png 000015.png 000017.png 000019.png
(662) $ file referee/000000.png
referee/000000.png: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 145x97, frames 3