Search code examples
prometheusgrafanaalertamazon-cloudwatch

Cloudwatch exporter (Prometheus datasource) with difference when getting metrics

Is there any configuration of cloudwatch exporter (Prometheus datasource) where the metrics have the least possible difference compared to Cloudwatch (datasource)?

Cloudwatch Exporter Config:

region: us-east-1
period_seconds: 30
delay_seconds: 15

metrics:

- aws_dimensions:
  - FunctionName
  - Resource
  aws_metric_name: Invocations
  aws_namespace: AWS/Lambda
  aws_statistics:
  - Sum

- aws_dimensions:
  - FunctionName
  - Resource
  aws_metric_name: Duration
  aws_namespace: AWS/Lambda
  aws_statistics:
  - Average
  - Maximum
  - Minimum

  - aws_dimensions:
  - FunctionName
  - Resource
  aws_metric_name: Errors
  aws_namespace: AWS/Lambda
  aws_statistics:
  - Sum

Last 30 minutes. Using Cloudwatch datasource.

Last 30 minutes. Using prometheus datasource (Cloudwatch Exporter).

enter image description here

Solution

  • No (but it can be better). You already collect 30sec. aggregated data, so there can be difference. You have also optimistic config (delay 15 sec). I found that metrics in the CloudWatch can have 1-2minutes delay.

    I would recommend Yace with this config options:

    scraping-interval: 60

metrics:
period: 60
length: 300
addCloudwatchTimestamp: true

    Check Yace doc for all config options and syntax.