How to filter the top metrics when the limit of allowed metrics are crossed on Cloudwatch?

I'm sending a metric to Cloudwatch when a server is connecting on my email server. That way, I can track who is connecting the most and ensure no one is causing any issues (like sending massive spam).

From that, I've created a graph to show the top most connecting servers.

The problem I'm facing is I now have more than 1500 different server name that has connected to my server, so the graph is now displaying a "Maximum number of metrics allowed".

For the graph, I used two queries:

e1 = FILL(SEARCH('MyMetric', 'Maximum', 900), 0) e2 (the one displayed) = SLICE(SORT(e1, MAX, DESC), 0, 50)

Because of this, I can not clearly see any connecting server that are above 900 since I have too many servers to filter through and Cloudwatch is restricting me.

How can I filter/query the data set from Cloudwatch to display properly that graph?

Note that I've already filtered the servers and only send to Cloudwatch those that have connected more than 250 times (so the small ones connecting a few time are excluded).

Solution

This sounds like an ideal use case for CloudWatch Contributor Insights. An excerpt from that page (emphasis my own):

You can use Contributor Insights to analyze log data and create time series that display contributor data. You can see metrics about the top-N contributors, the total number of unique contributors, and their usage. This helps you find top talkers and understand who or what is impacting system performance. For example, you can find bad hosts, identify the heaviest network users, or find the URLs that generate the most errors.

It might actually end up giving you more coverage (you won't need to drop those with <250 connections) and being simpler, because your app only needs to generate logs instead of calling PutMetricData. It will likely end up cheaper too. The general idea is that do the following:

Change your code to write a log line each time someone sends an email through your server. I am guessing it might look like this:

{"clientIp": "1.2.3.4", "emailSizeInBytes": 789, "recipientCount": 17}

Create a CW Contributor Insights rule. The console has a wizard that helps, but the raw rule syntax will look a bit like this:

{
  "Schema": {
    "Name": "CloudWatchLogRule",
    "Version": 1
  },
  "LogGroupNames": [
    "YourAppLogGroup"
  ],
  "LogFormat": "JSON",
  "Contribution": {
    "Keys": [
      "$.clientIp"
    ],
    "ValueOf": "$.emailSizeInBytes"
  },
  "AggregateOn": "Sum"
}

This will create a rule that sums together all the bytes sent by each client IP. You could create a second rule that also calculates which client IPs send emails to the most recipients by changing ValueOf to $.recipientCount. Or a third rule that counts only the number of emails sent, rather than the cumulative size, by changing AggregateOn to Count instead of Sum.