I'm using Datadog's statsd client to record the duration of a certain server response. I used to pass in quite a few number of custom tags when time-ing these responses. So I'm in the process of reducing the number of custom tags.
However, the problem is that when I reduce the number of tags passed in, there is extra latency of server response, which isn't intuitive because I'm passing in fewer tags and the implementation hasn't changed.
According to Datadog and Etsy (which originally released statsd), these methods that record these metrics aren't blocking. However, they must be using some extra threads to perform this.
What could be the issue? Are there possible any side effects associated with using this client?
I found the answer on Datadog's help forum: "How to graph percentiles in Datadog".
- Making a change to increase tag complexity (adding additional tags to be more specific) will lead to changes in the behavior of a rolled up metric visualization
- EX: Whereas before the change METRIC_NAME.avg (without any tags) would be aggregating across all raw points (statsd takes all the raw datapoints, aggregates it and then ships over a single metric stream), adding a tag like region (US, EU) tag causes statsd to bin raw datapoints into two region bins, aggregate them, and ship over two streams. This means when graphing METRIC_NAME.avg AVG by * means an aggregate across the two streams rather than a single one
So the gist is that the latency itself didn't go up, but aggregating over multiple streams (where each stream corresponds to each custom tag) caused the graph to display a different shape.