Search code examples
proxyipip-addressweb-crawlertor

How do I get a collection (or a stream) of distinct IP addresses using Tor?

I'm writing a web crawler for academic research. This crawler makes millions of requests that I want to distribute over ten or so IP addresses.

My machine has one IP address already. I can get a second by using Tor. Can I get even more IP addresses out of Tor? Here are ideas (and questions surrounding them) that I have for doing this

  • Run multiple instances of Tor; each provides an IP address. (But will Tor map more than one or two anonymized IP addresses to my machine?)
  • Run one instance but for each request change its identity. (But will Tor rate-limit this behavior, as mentioned here?)

Would either of these ideas work, or do the bits in parentheses make them fail? Any other ideas?

Solution

  • Tor relays have rate limits. NEWNYM is limited to 5 second intervals.

    If they're not fast enough, a willing botnet or app engine should work.