Search code examples
web-scrapingweb-crawlerpython-3.7http-status-code-403

web scraping / web crawling showing 403 error on the site i want to crawl

import requests
from bs4 import BeautifulSoup
url ='https://www.vesselfinder.com/vessels'
headers= {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)'}
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
response.status_code

i tried different user agent but still not working, i tried other sites it work but this website not working, help me to crawl all vessel data from this site. thanks in advance!!!

Solution

  • Server wants an additional header for language

    import requests

headers = {
    'user-agent': 'Mozilla/5.0',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
}

response = requests.get('https://www.vesselfinder.com/vessels', headers=headers)
response.status_code