I'm trying to use AWS CloudWatch Logs Insights to count the number of requests coming from each IP address within 5 minutes of an API.
Here's an example of the data I want to get.
Time Range: 01:05 - 01:10
| Request Source | Number of Requests |
|---|---|
| 53.240.51.81 | 314 |
| 36.241.227.134 | 237 |
| 201.72.45.51 | 201 |
Time Range: 01:10 - 01:15
| Request Source | Number of Requests |
|---|---|
| 43.142.151.81 | 334 |
| 36.131.237.174 | 230 |
| 161.72.15.52 | 198 |
Time Range: 01:15 - 01:20
| Request Source | Number of Requests |
|---|---|
| 31.132.141.91 | 334 |
| 39.138.217.174 | 230 |
| 191.76.15.42 | 198 |
...
And here's my query:
fields @timestamp, @message
| filter (@message like "/my_api")
| parse @message '* - * [*] * * * "*" * * * "*" "*" "*"' as remote_addr, remote_user, time_local, http_method, path, http_version, header, status_code, request_length, body_bytes_sent, http_referer, http_user_agent, http_x_forwarded_for
| stats count() as requestCount by http_x_forwarded_for
| sort requestCount desc
The problem of the query above is that it only counts the number of requests within the time range I specified.
So if I want to get the data during the past 24 hours, I'll have to specify the time range of Logs Insights for 24 * 60 / 5 = 288 times. And run the query for 288 times.
It would be very time-consuming to do the steps I describe above.
The reason I want to get the data is that I want to use AWS WAF to set a rate limit on my API.
Before I apply the rate limit, I need to know the request rate of my API during normal days.
Tried out a similar query against my CloudTrail events log, and bin() is what you want:
fields @timestamp, @message
| stats count(*) by eventName, bin(5m)
| sort desc
| limit 20