Search code examples
pythonpython-requestspython-multithreadinghttplibhttplib2

HTTP requests using multiple IP addresses on python

I'm writing a python script that will send http requests concurrently to the urls mentioned in a file using python. The script works fine for a single IP address. The OS I'm using is linux. I've generated virtual IP addresses like eth0:1,eth0:2 etc. I want to send HTTP requests using these virtual IP addresses along with the eth0 IP address concurrently. I use the requests module for http requests and threading module for concurrent requests. Kindly help me. I'm trying to develop a web testing tool.

Solution

  • I think you wanted to avoid "Crawl Delay" and do faster crawl on one server!

    In this case, Remote Web Server will recognize request from only one IP!!

    I think using parallel + curl + python script is more simple and best way.

    or use https://pypi.python.org/pypi/pyparallelcurl/0.0.4

    or use a lot of servers.

    and refer to https://code.google.com/p/httplib2/issues/detail?id=91