Search code examples
pythonmechanicalsoup

Using MechanicalSoup behind proxy

I am trying to build a simple webbot in Python, on Windows, using MechanicalSoup. Unfortunately, I am sitting behind a (company-enforced) proxy. I could not find a way to provide a proxy to MechanicalSoup. Is there such an option at all? If not, what are my alternatives?

EDIT: Following Eytan's hint, I added proxies and verify to my code, which got me a step further, but I still cannot submit a form:

import mechanicalsoup

proxies = {
    'https': 'my.https.proxy:8080',
    'http':  'my.http.proxy:8080'
}
url = 'https://stackoverflow.com/'
browser = mechanicalsoup.StatefulBrowser()
front_page = browser.open(url, proxies=proxies, verify=False)
form = browser.select_form('form[action="/search"]')
form.print_summary()
form["q"] = "MechanicalSoup"
form.print_summary()
browser.submit(form, url=url)

The code hangs in the last line, and submitdoesn't accept proxies as an argument.

Solution

  • It seems that proxies have to be specified on the session level. Then they are not required in browser.open and submitting the form also works:

    import mechanicalsoup

proxies = {
    'https': 'my.https.proxy:8080',
    'http':  'my.http.proxy:8080'
}
url = 'https://stackoverflow.com/'
browser = mechanicalsoup.StatefulBrowser()
browser.session.proxies = proxies   # THIS IS THE SOLUTION!
front_page = browser.open(url, verify=False)
form = browser.select_form('form[action="/search"]')
form["q"] = "MechanicalSoup"
result = browser.submit(form, url=url)
result.status_code

    returns 200 (i.e. "OK").