How to stream AWS EC2 cloud-init logs to ELK?

Question

My app dynamically creates EC2 instances via the AWS EC2 Javascript SDK. With each creation, a Bash script is run via the UserData parameter in the runInstances method. The logging for this Bash script appears in /var/log/cloud-init-output.log on the instance. Currently, I have to manually log in to the instance to review this file, which is tedious and not scalable.

How do I stream this logging to an outside destination? I want to use the ELK stack, but the specific details of my logging solution probably aren't as important as just figuring out how to redirect the cloud-init logging.

Answer 1

I've come up with several methods.

---

Directly stream a log with Logstash

In this way, you need to install Logstash on your EC2 instances. But you can stream log to anywhere you want, such like Amazon ES, ElasticCloud or your own ElasticSearch host.

1. Install Logstash

• This can be done by either scripting in UserData or creating custom AMI with Logstash installed.
• Logstash is available via binary file, package manager and docker image.

2. Configure

• Logstash config file would be like:

input {
  file {
    path => ["/var/log/cloud-init-output.log"] # specify log file path here
  }
}

# if you need to process log on the fly
# you can use filter plugin
# filter {}

output {
  elasticsearch {
    hosts => ["https://YOUR_ES_HOST"]
    index => "YOUR_INDEX_NAME"
    user => "YOUR_USERNAME"
    password => "YOUR_PASSWORD"
  }
}

3. Start Logstash service

---

Use CloudWatch and Amazon ES

Less time for implementation but Amazon ES is required.

1. Push log to CloudWatch Logs with CloudWatch log agent.

• Configuration can be found in Logs Section on this page.
• Be sure to set /var/log/cloud-init-output.log at file_path field.

2. Create ElasticSearch subscription filter in CloudWatch Logs.

• See here.

---

Of course there are lots of way to archive that, these two are relatively simple imo.