I want to use RabbitMQ with StormCrawler. I already saw that there is a repository for using RabbitMQ with Storm: https://github.com/ppat/storm-rabbitmq
How would you use this for the StormCrawler? I would like to use the Producer as well as the consumer.
For the consumer there seems to be some documentation. What about the Producer? Can you just put the config entries in the storm crawler config or would I need to change the source code of the RabbitMQProducer?
You'd want the bolt which sends URLs to RabbitMQ to extend AbstractStatusUpdaterBolt as the super class does a lot of useful things under the bonnet, which means that you would not use the Producer out of the box but will need to write some custom code.
Unless you are certain that there will be no duplicates URLs, you'll need to deduplicate the URLs before sending them to the queues anyway, which could be done e.g. with Redis within your custom status updater.