Search code examples
scrapyscrapy-pipeline

scrapy 307 redirects to same page

I am trying to scrape product details from jabong and the script is running fine when I scrape one URL but it redirects to 307 when I use more than one urls in yield request

i tried private proxies, user agents, disabling cookies, enabling dont_merge cookies

this is what i m getting:

2019-05-20 03:41:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.jabong.com/Hangup-Green-Solid-Nehru-Jacket-7282330.
htm> from <GET https://www.jabong.com/Hangup-Green-Solid-Nehru-Jacket-7282330.htm>
Solution

  • Yes, they do the same thing to Chrome and it is very likely because you have cookies disabled.

    You'll need to spend the time to find out which cookie they care about, and whether using that same cookie value compromises the anonymity of your crawl. That's too time consuming for a SO answer but it should be straightforward to do.