Search code examples
mesosamazon-cloudwatchmesosphereprometheus

Setting up cloud watch exporter for prometheus on mesosphere DCOS cluster

I have set up cloud watch exporter for Prometheus on my AWS mesosphere DCOS cluster. I have 'CloudWatchFullAccess' policy enabled. But meter, 'cloudwatch_exporter_scrape_error' shows non-zero value. I want to know why the scrape is giving error.

Where can I check logs or how to debug this problem?

Also the config file I am using is 

{
   "region": "ap-southeast-1",
   "metrics": [
        {"aws_namespace": "AWS/ELB", "aws_metric_name": "HealthyHostCount",
         "aws_dimensions": ["AvailabilityZone", "LoadBalancerName"],
         "aws_dimension_select": {"LoadBalancerName": ["name of my LB"]},
         "aws_statistics": ["Sum"]
        }
      ]
}

but I do not any meters in addition to meters - cloudwatch_requests_total, cloudwatch_exporter_scrape_duration_seconds, cloudwatch_exporter_scrape_error exposed to Prometheus.

How to get additional meters from cloudwatch_exporter?

Solution

  • It looks as though you're trying to use an IAM instance profile, but you can't access http://169.254.169.254. This is some form of problem with your network setup, as this should work out of the box on EC2.

    You have two options.

    1. You can fix your network setup so you can get to 169.254.169.254 again.
    2. Or you could create an IAM user with the cloudwatch:ListMetrics and cloudwatch:GetMetricStatistics IAM permissions, and generate accesskeys and put them in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or ~/.aws/credentials.

    See https://github.com/prometheus/cloudwatch_exporter#credentials-and-permissions