Search code examples
apache-stormstormcrawler

Can I Increase the Workers during the crawl Processs


Working on Storm Crawler 1.10 and Apache storm 1.2.2. How can I change the workers and number fetch threads while crawl process is going on.


Solution

  • The number of fetch threads can't be changed on the fly, same for the number of workers. You can simply stop the topology, change the config and restart it. The interruption will be minimal and the crawl we pick up where it left