Search code examples
pythonhttp-proxy

make any internet-accessing python code work (proxy + custom .crt)


The situation

If the following is not done, all outgoing HTTP or HTTPS requests made with python ends in a WinError 10054 Connection Reset, or a SSL bad handshake error.

  1. set the HTTP_PROXY, HTTPS_PROXY environment variable, or their counterparts
  2. What needs to be verified must be verified with a custom .crt file.

For example, assuming the .crt file is in place, both gets me a 200 OK:

import os
os.environ['HTTP_PROXY'] = #some_appropriate_address
os.environ['HTTPS_PROXY'] = #some appropriate_address
requests.get('http://www.google.com',verify="C:\the_file.crt") # 200 OK
requests.get('http://httpbin.org',verify=False) # 200 OK, but unsafe
requests.get('http://httpbin.org') # SSL bad handshake error

The Problem

There is this massive jumble of pre-written code (heavily utilizing urllib3 and requests and possibly other pieces of internet-accessing code) I have, and I have to make it work under the conditions outlined above.

Sure, I can write verify='C:\the_file.crt' for every requests.get(), but that can very quickly get hairy, right? And the code may also be using some other library (that is not requests). So I am looking for a global setting (environment variable etc.) I should alter, so that everything works well (return a 200 OK upon a GET request to a server, whether or not the code is written in requests-py).

Also, if there is no such way, I would like an explanation as to why.

What I tried (am trying)

Maybe editing the .condarc file (via conda --config) is a solution. I tried, to no avail: python gives me a "SSL verification failed" error. On the contrary, note that the code snippet above gave me a 200 OK. To my knowledge, this does not fit nicely with many situations that were previously discussed in Stack Overflow.

By the way, setting ssl_verify to false does not solve the problem either; I still get a bad handshake error for some reason.

I am using Win 10, Python 3.7.4 (Anaconda).

Update

I have edited the question to prevent future misunderstandings about the content of this question. A few answers below are a reiteration of what was written here from the start.

The current answers are not entirely satisfactory either, as they only seem to address the case where I am using requests or urllib3.


Solution

  • You should be able to get any python code that uses the requests module(which is inside urllib3) to work behind a proxy without modifying the python code itself by setting the following environment variables in Windows.

    http_proxy           http://[<user>:<pwd>@]<http_host>:<http_port>  
    https_proxy          http://[<user>:<pwd>@]<https_host>:<https_port>
    requests_ca_bundle   <path_to_ca_bundle.crt>
    curl_ca_bundle       <path_to_ca_bundle.crt>
    

    You can set environment variables by doing the following:

    • Press Windows-Key + R, enter sysdm.cpl ,3 (mind the space before the comma) and press Enter
    • Click the Environment variables button
    • In either of the fields (User variables or System variables), add the four variables