Issues Sending Kafka and Zookeeper Metrics to AWS CloudWatch Using JMX and CloudWatch Agent

Question

I'm trying to send metrics from Kafka and Zookeeper to AWS CloudWatch using the CloudWatch Agent and JMX. I've configured JMX for both Kafka and Zookeeper and set up the CloudWatch Agent configuration file. However, I'm encountering issues with the configuration validation.

Here are the steps I've followed:

Configured JMX for Kafka and Zookeeper:

For Kafka:

export KAFKA_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

For Zookeeper, I used the default port 9998.

CloudWatch Agent Configuration File: Here is the content of my amazon-cloudwatch-agent.json file:

{
  "agent": {
    "metrics_collection_interval": 60,
    "region": "us-east-1"
  },
  "metrics": {
    "namespace": "KZMetrics",
    "metrics_collected": {
      "jmx": {
        "kafka": {
          "jmx_endpoint": "localhost:9999",
          "measurement": [
            {
              "name": "java.lang:type=OperatingSystem/SystemCpuLoad",
              "alias": "Kafka_CPUUsage",
              "unit": "Percent"
            },
            {
              "name": "java.lang:type=Memory/HeapMemoryUsage",
              "alias": "Kafka_MemoryUsage",
              "unit": "Bytes"
            }
          ],
          "metrics_collection_interval": 60
        }
      }
    }
  }
}

Command to Start CloudWatch Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Error Encountered: The command runs but I get the following validation errors:

2024/08/25 08:10:45 E! Invalid Json input schema.
2024/08/25 08:10:45 E! Invalid Json input schema.
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Must validate one and only one schema (oneOf)
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Must validate at least one schema (anyOf)
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : jvm is required
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : endpoint is required
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Additional property jmx_endpoint is not allowed

Agent Status:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
{
  "status": "running",
  "starttime": "2024-08-25T07:42:10+00:00",
  "configstatus": "configured",
  "version": "1.300043.0b781"
}

It seems like there are issues with the JSON schema or parameters. I’ve reviewed the documentation, but I’m having trouble understanding it or finding the correct details. Can someone help me pinpoint what might be wrong with the configuration or suggest a solution?

Thanks in advance!

Answer 1

After encountering JSON schema validation issues with the CloudWatch Agent configuration file, I switched to using collectd to send Kafka and Zookeeper metrics to AWS CloudWatch. I followed the guidance in this AWS article, which worked well for my use case.