Search code examples
pythonpython-3.xweb-scrapingpython-requestspython-requests-html

Requests library in python keep working and don't response anything

I try to send a request to a website, but after I run my code, it's running forever and getting no response. Please someone help me, here is my code:

import requests

req = requests.session()
url = "https://www.stepstone.de/"
response = req.get(url)
print(response.status_code)

I try put some headers in it, but it doesn't work for me. If you have any advice, I would appreciate it if you could share it with me.

Solution

  • By setting User-Agent header I was able to obtain response from the server:

    import requests

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0"
}

url = "https://www.stepstone.de/"

html_text = requests.get(url, headers=headers).text
print(html_text)

    Prints:

    <!DOCTYPE html>
<html lang="de">
  <head>
    
    <title>Jobs sind unser Job | Stepstone</title>
    
    <script type="application/ld+json">
    {
      "@context": "https://schema.org",
      "@type": "Organization",
      "url": "https://www.stepstone.de",
      "logo": "https://www.stepstone.de/sharedassets/logo/core/stepstone/logo.svg"
     }
    </script> 
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta charset="utf-8" />

...