I am writing a crawler in Python that will run through Tor. I have Tor working and used code from this YouTube tutorial on how to route my Python requests to go through the Tor SOCKS proxy at 127.0.0.1:9050
.
What I can't figure out is how to toggle this on/off within my script. Some requests I want to go through Tor and some I don't. Basically, I can't figure out the correct "close" or "shutdown" method in the socket
objects I am using because I don't understand them.
Here's what happens now
import socket
import socks
import requests
def connect_to_socks():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket
r = requests.get('http://wtfismyip.com/text')
print r.text #prints my ordinary IP address
connect_to_socks()
r = requests.get('http://wtfismyip.com/text')
print r.text #prints my Tor IP address
How do I turn off the socket routing to the SOCKS proxy so that it goes through my ordinary internet connection?
I'm hoping to use requests
instead of urllib2
as it seems a lot easier but if I have to get into the guts of urllib2
or even httplib
I will. But would prefer not to.
Figured it out by listening to this good YouTube tutorial.
Just need to call socket.setdefaultproxy()
and it brings me back.