Search code examples
microserviceshaproxymonitor

How to monitor connection in local network

I have a ton of services: Node(s), MySQL(s), Redis(s), Elastic(s)...
I want to monitor how they connect to each other: Connection rate, Number alive connection... (Node1 create 30 connection to Node2/MySQL/Redis per second...) like Haproxy stat image attached below.

Haproxy stat

Currently i have two options:

  • Haproxy (proxy): I want to use single service Haproxy to archive this but it's seem very hard to use ALC detect what connection need forward to what service.
  • ELK (log center): I need to create log files on each service (Node, MySQL, Redis...) and then show them on the log center. I see that a ton of works to do that without built-in feature like Haproxy stat page.

How to do this? Is log center good in this case?

Solution

  • The problem

    I think your problem is not collecting and pipelining the statistics to Elasticsearch, but instead the ton of work extracting metrics from your services because most of them do not have metric files/logs.

    You'd then need to export them with some custom script, log them and capture it with filebeat, stream to a logstash for text processing and metric extraction so they are indexed in a way you can do some sort of analytics, and then send it to elasticsearch.

    My take on the answer

    At least for the 3 services you've referenced, there are Prometheus exporters readily available and you can find them here. The exporters are simple processes that will query your services native statistics APIs and expose a prometheus metric API for Prometheus to Scrape (poll).

    After you have Prometheus scraping the metrics, you can display them in dashboards via Grafana (which is the de facto visualization layer for Prometheus) or bulk export your metrics to wherever you want (Elasticsearch, etc..) for visualization and exploration.

    Conclusion

    The benefits of this approach:

    1. Prometheus can auto-discover new nodes you add to your networks
    2. Readily available exporters from haproxy, redis and mysql for Prometheus
    3. No code needed, each exporter requires minimal configuration specific to each monitored technology, it can easily be containerized and deployed if your environment is container oriented, otherwise you just need to run each exporter in the correct machines
    4. Prometheus is very, very easy to deploy