Search code examples
pythonweb-scrapingscrapy

Scrapy: saving cookies between invocations


Is there a way to preserve cookies between invocations of a scrapy crawler? The purpose - the site requires log in, and then maintains the session via cookies. I'd rather reuse the session than re-login every time.


Solution

  • Please refer to the docs about cookies.

    FAQ entry

    CookiesMiddleware

    Alternatively you can send Request objects with cookies managed by yourself (you can read cookies from Response objects' head).

    About Request and Response objects