Search code examples
javaweb-crawlerstormcrawler

StormCrawler do action when crawling one domain finished

I want to do an action (in my case send a tuple to one bolt) when the crawler finished crawling one domain.

I saw that StormCrawler is able to do even revisit a website after a given interval. In which component or how could I see when one domain is finished crawling in the scenario of crawling multiple domains simultaneously?

My current setup is using StormCrawler with Elasticsearch and Kibana.

Solution

  • I implemented a custom Bolt which follows the former approach but could also be extended or modified for the second approach mentioned in the answer of Julien Nioche