Search code examples
pythonhtmlpython-3.xbeautifulsouphtml-parsing

BeautifulSoup HTML getting src link


I'm making a small web crawler using python 3.5.1 and requests module, which downloads all comics from a specific website.I'm experimenting with one page. I parse the page using BeautifulSoup4 like below:

import webbrowser
import sys
import requests
import re
import bs4

res = requests.get('http://mangapark.me/manga/berserk/s5/c342')
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')

for link in soup.find_all("a", class_ = "img-link"):
    if(link):
        print(link)
    else:
        print('ERROR')

When I do print(link) there are the correct HTML parts I'm interested in, but when I try to get only the link in src using link.get('src') it only prints None.

I tried getting the link using:

img = soup.find("img")["src"]

and it was OK, but I want to have all the src links, not the first link. I have little experience with beautifulSoup. Please point out what's going on. Thank you.

The sample HTML part from the website I'm interested in is:

<a class="img-link" href="#img2">
    <img id="img-1" class="img"
          rel="1" i="1" e="0" z="1" 
          title="Berserk ch.342 page 1" src="http://2.p.mpcdn.net/352582/687224/1.jpg"
          width="960" _width="818" _heighth="1189"/>        
</a>

Solution

  • I would do it in one go using a CSS selector:

    for img in soup.select("a.img-link img[src]"):
        print(img["src"])
    

    Here, we are getting all of the img elements having an src attribute located under an a element with a img-link class. It prints:

    http://2.p.mpcdn.net/352582/687224/1.jpg
    http://2.p.mpcdn.net/352582/687224/2.jpg
    http://2.p.mpcdn.net/352582/687224/3.jpg
    http://2.p.mpcdn.net/352582/687224/4.jpg
    ...
    http://2.p.mpcdn.net/352582/687224/20.jpg
    

    If you still want to use the find_all(), you would have to nest it:

    for link in soup.find_all("a", class_ = "img-link"):
        for img in link.find_all("a", src=True):  # searching for img with src attribute
            print(img["src"])