There is How to pass all Python's traffics through a http proxy? However that one does not deal with sock proxy. I want to use sock proxy which we can get easily with ssh tunneling.
ssh -D 5005 user@server
Dealing with socks proxy is not straight forward like dealing with http proxy just setting the environment variables. There are three approaches to use socks proxy.
PySock
We need to install it with pip install pysocks
. Then we can set up the following code in our python code.
import socks
import socket
# Define your SOCKS proxy
proxy_host = 'localhost'
proxy_port = 5005
# Set up the proxy
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, proxy_host, proxy_port)
socket.socket = socks.socksocket # Monkey-patch socket creation to use SOCKS proxy
response = requests.get('https://ifconfig.me/')
print("Response from https://ifconfig.me/:", response.text)
tinyproxy
However not all the python code, especially some third party library will not use that PySock proxy set up. Though those third party library will honor the http proxy. When that happens, we can use easy to set up tinyproxy
with the following configuration in tinyproxy.conf
. (I will leave installing and running as service steps in the reader capable hand.)
Port 8888 # or any available port
Listen 127.0.0.1
upstream socks5 127.0.0.1:5005
Now we have a local http proxy server which make use of the sock proxy from ssh tunneling and we just need to apply the http proxy solution here.
proxy = 'http://localhost:8888'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy
response = requests.get('https://ifconfig.me/')
print("Response from https://ifconfig.me/:", response.text)
sshuttle
This approach is not much different from using VPN any more but it resonates with the question where the original purpose is to make use of the ssh tunneling. So this one is just doing the better tunneling without doing the original question ssh -D 5005 user@my_server.com
. You need root access on your local development machine but you don't need such access in ssh server. You just install it using aptget
or brew
... Then run the following:
sshuttle -r user@my_server.com 0.0.0.0/0 -x my_server.com
curl https://ifconfig.me/
-r means remote server
0.0.0.0/0 means for all the ip address range to go through this
-x means to exclude (this host to prevent cyclic loop)
If you need to use sshuttle with .key, .csr or .pem files for authentication
sshuttle -r user@my_server.com 0.0.0.0/0 -x my_server.com --ssh-cmd 'ssh -i /your/key/path.pem'
With this, you don't even need to set the environment variables about the http proxy since the whole system goes through that sshuttle
pipe.
I personally prefer the tinyproxy
approach since it does not mess up with the project set up and code. It is just the environment setting only. So in the end, the http proxy with tinyproxy is the preferable solution.