Search code examples
pythonweb-scrapinghttp-status-code-404

get Links corresponding to error codes 404


I have approximately 500 player Links. Apart from two, all the links return the response code 200. However, I need exact links / urls that have the code 404.

for p in range(len(playerLinks)):
    
       
    page = playerLinks[p]
    response = requests.get(page, headers={'User-Agent': 'Custom5'})
    print(response.status_code)
    injury = response.text
    soup = BeautifulSoup(injury, 'html.parser')
    table = soup.find(id="yw1")

thank you very much for your support. An example for a player Links(how those links look like) is this (but there are at least 500 different)

https://www.transfermarkt.de/Timothy Fosu-Mensah/verletzungen/spieler/315131']

Solution

  • You can do that with simple if statement.

    for p in range(len(playerLinks)):
        
           
        page = playerLinks[p]
        response = requests.get(page, headers={'User-Agent': 'Custom5'})
    
        if response.status_code == 404:
            print(page)
    
        injury = response.text
        soup = BeautifulSoup(injury, 'html.parser')
        table = soup.find(id="yw1")