Search code examples
javascriptweb-crawlerapify

How to stop Apify from saving handled requests?


It seems Apify is saving handled requests in 'handled' directory. I want to stop this because apify won't crawl the page I've crawled before. Anyone know how to stop Apify from saving handled request?


Solution

  • You can work around this by putting the request in queue multiple times with a different uniqueKey so that it is not automatically filtered out by request queue after already having been processed once. Local files only simulate the behavior of RequestQueue exactly as it behaves on the Apify platform.