deploy bolts/spout to a specific supervisor

Question

We are running a storm application using a single type on instance in AWS and a single topology to run our system.

This is causing some resource limitation issues. The way we want to address this is by splitting our IO intense bolts into a cluster of a few dozens t1.small machines (for example) and all our CPU intense bolts to two large machines with lots of cpu & memory.

Basically what i am asking is, is there a way to start all this supervisors and then deploy one topology that include cpu intense bolts on the big machines and to the small machines the deploy IO bolts?

Answer 1

You can implement a custom scheduler using interface IScheduler.

See

• http://www.exogeni.net/2015/04/enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
• https://dcvan24.wordpress.com/2015/04/07/metadata-aware-custom-scheduler-in-storm/
• https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/DemoScheduler.java