I have integrated my springboot application with dynatrace. And able to send application metrics and cassandra db metrics to dynatrace datastore. Able to view cassandra db queries, response time of each database call e.t.c.,.
How dynatrace is fetching this information? I want to fetch similar information in grafana via prometheus. So, just thought of checking, how dynatrace works internally. So, I could implement the same in grafana.
Can anyone please suggest me out regarding the same ?
I believe that Dynatrace uses a dedicated agent to scrape Cassandra metrics, but you shouldn't be able to use it for Prometheus.
However, there are some tried and true solutions out there to expose Cassandra metrics to Prometheus:
Both are relatively easy to set up and use, and there are plenty of publicly available Cassandra dashboard templates for both in Grafana.
Other than these two, I've only heard of Instaclustr's Cassandra exporter, but never got to try it.
I personally have used JMX Exporter the most with Prometheus and Grafana and recommend it for most use cases. As long as you expose only exactly what you need for your dashboards (configured in the JMX Exporter yaml file) and tune the scrape interval in the Prometheus configuration, the CPU overhead should be negligible.
For query traces, specifically, I don't think that neither of the options I suggested can do it, since they rely on MBeans to expose metrics, and as far as I know, there is no MBean for query traces. Either way, I'd guess that exposing query traces is rather expensive, specially considering the high request throughput that most Cassandra clusters can handle.