Search code examples
pythonurllibhttp-status-code-405

python urllib randomly appearing HTTP Error 405:

Hello knowledgeable people,

I'm writing a code to grab some information from some subpages of a specific website.

The following three lines of code are working in around 8 of 10 cases (while using the same link/URL which is as well posted). But in two out of ten cases I'm getting the error 'HTTPError: HTTP Error 405: '.

I have no clue why the webpage is rejecting my request sometimes and sometimes not. And even more important: How may I rebuild this code to get the answer for sure without an error message? (even if it takes more time)

import urllib

link = 'https://www.immobilienscout24.de/expose/128433102?utm_medium=email&utm_source=system&utm_campaign=fulfillment_update&savedSearchId=116004747&immoTypeId=2&PID=113689187&referrer=ff_listing&utm_content=fulfillment_item#/'

f = urllib.request.urlopen(link)
Solution

  • In the meantime I solved it myself:

    1. the request within a "try"
    2. a dict in which I write for each url if it worked or not
    3. a while-loop in which I'm repeating it for the url for which it did not work the previos times.
    4. and a break after there is no url left without a successful web scrabbing.
    5. In between I'm transforming the data in the way I need it.

    Probably not a extraordinary tactic.

    Today it took 9 rounds in the loop before 17 links all have been srabbed.

    I'm still curious about the reason of the failure and a direct solution. But my solution gets the data I want without any problems.