In microservices architecture, I'd like to detect and generate an alert based on threshold when a service goes down. I was wondering to use for each client-side microservice the circuit breaker to send informations when the circuit switches and to create an alert related to the state 'down' of the target. But I don't if it's a good pattern. Moreover I have two concerns, the first one is to monitor microservices and aggregate their data to generate an alert if threshold is reached and the second one is to do the same thing with third-party services (external services) used by microservices by a gateway. According to you, What's the best way to monitor micro-services ? Thanks
A circuit breaker should be used when you need to allow a failing service to recover, instead of continuously hitting it with requests, despite the fact that it cannot serve them. This is a nice article that you can check for more details about how to use this pattern.
So if your service is likely to recover in a short amount of time, you can use a circuit breaker.
Otherwise, which I think it's the case here (because you want an alert to be fired when the service is down), I would focus more on the reasons why that microservice would fail, and I would try to minimize the chance of occurrence.
If you use Kubernetes, you can monitor the microservices using Prometheus. Here is a nice article that can get you started. Prometheus scrapes the Kubernetes API and exposes metrics related to pods. You can create an alert based on those.
To monitor external services, you can use Prometheus for this too. If the external service is able to expose metrics via Prometheus, then you just plug it and you're done. Otherwise, you have to write some code to check the health of that service and expose a metric based on that.