Search code examples
pythonproxyweb-scrapingrequestscrapy

Check IP address used for a request Python/Scrapy + ProxyMesh

I started to use ProxyMesh with Scrapy. ProxyMesh is supposed to provide a pool of IP addresses on one port. The sample code code below is being repeated in a loop, middleware is enabled and overall this works fine. Can I track (and if so - how?) which IP address is being used for each specific request? 

request = scrapy.Request(producturl, self.parse_product_info)
request.meta['proxy'] = 'http://uk.proxymesh.com:xxx'
yield request

I found similar posts on SOF, but not addressing this specific question.

Solution

  • Like specified in the comments, the information comes on the response headers, just check it:

    def parse_response(self, response):
    print response.headers

    You should see the X-Proxymesh-Ip header with the assigned proxy.

    Another alternative could be to use crawlera which offers even more features (like headers, sessions and cookie handling) and better documentation.