Hystrix to open circuit for problematic service instances?

Question

Assume the following scenario:

1. I have ServiceA calling ServiceB with Hystrix enabled
2. There are 2 instances of ServiceB
3. One of the instances is facing some pretty serious issues where its endpoint is consistently throwing errors, but the other instance is doing OK
4. errorThresholdPercentage is set to 80%
5. Given that we have one service instance that's doing fine, it's not likely that errorThresholdPercentage would ever go above 80% (at most it would go up to 50% only)
6. That means that as long as the problematic instance is running, we would still be invoking it and getting errors

Is there any way to configure Hystrix to flip open the circuit down to the instance level?

Answer 1

There is no way to configure Hystrix circuit breaker for the instance level.

Instead, you can achieve it by using Ribbon with Hystrix. There is a IRule called AvailabilityFilteringRule that is used as default. It has its own circuit breaker that is not related to hystrix. It will filter out the problematic instance from the instance list during certain seconds. You can adjust this behavior with the below ribbon properties.

niws.loadbalancer.default.circuitTripTimeoutFactorSeconds (default 10secs)
niws.loadbalancer.default.connectionFailureCountThreshold  (default 3 failures)
niws.loadbalancer.default.circuitTripMaxTimeoutSeconds (default 30secs)

By using ribbon + hystrix, you can achieve two levels of circuit breaker like below.

• service level circuit breaker - hystrix
• instance level circuit breaker - Ribbon (by AvailabilityFilteringRule)

You can find some description about this here