StatsD, has been around for some years now, thanks Etsy and Flickr. I have recently stumbled upon it and been 'playing' with it. There are several reasons that make me love it.
I wonder if somebody is using it along large and heavily used systems and has some feedback on it? How is StatsD working out for your cases?
Statsd works well up to 20k packets/sec (UDP packets / sec), but then starts to drop metrics after that as it's not fast enough to process that many. For some metrics workloads, accuracy is required and so sampling is not an option. It can be pretty easy to eat up this 20k / sec budget.
There are various other statsd implementations that have better performance. One of them is https://github.com/github/brubeck which claims it can process up to 4 million metrics / sec. YMMV, but I've been using brubeck in production and it can handle way more load than statsd can.