Search code examples
elasticsearchwindows-server-2008logstashamazon-sqsamazon-iam

Windows Logstash output to AWS SQS with IAM EC2 Role "Unable to access SQS queue"


I feel like I'm missing something here as nobody on the Interwebs seems to be having any issues. I have a Windows Server 2008 R2 EC2 instance setup with an IAM Role that allows sqs:SendMessage. I successfully sent a message on the command line using the AWS CLI and the command aws sqs send-message. When I attempt to use Logstash-1.4.2 I get the following error:

Unable to access SQS queue 'Dev-ELK-LogstashBuffer': The specified queue does not exist or you do not have access to it. {:level=>:error}
Failed to flush outgoing items {:outgoing_count=>1, :exception=>#<NoMethodError: undefined method `batch_send' for nil:NilClass>, :backtrace=>
["C:/logstash/logstash-1.4.2/lib/logstash/outputs/sqs.rb:131:in `flush'", 
"C:/logstash/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", 
"org/jruby/RubyHash.java:1339:in `each'", 
"C:/logstash/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", 
"C:/logstash/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", 
"C:/logstash/logstash-1.4.2/lib/logstash/outputs/sqs.rb:123:in `receive'", 
"C:/logstash/logstash-1.4.2/lib/logstash/outputs/base.rb:86:in `handle'", 
"(eval):129:in `initialize'", 
"org/jruby/RubyProc.java:271:in `call'", 
"C:/logstash/logstash-1.4.2/lib/logstash/pipeline.rb:266:in `output'", 
"C:/logstash/logstash-1.4.2/lib/logstash/pipeline.rb:225:in `outputworker'", 
"C:/logstash/logstash-1.4.2/lib/logstash/pipeline.rb:152:in `start_outputs'"], 
:level=>:warn}

Here's my output configuration for logstash:

output {
    sqs {
        queue => "Dev-ELK-LogstashBuffer"
        region  => "us-west-2"
        use_ssl => true
    }

    stdout { codec => rubydebug }
}

I am able to use elasticsearch output directly but trying to add the message queue buffer between my servers and the elasticsearch cluster.

Thanks!


Solution

  • The correct IAM policy:

    {
        "Statement":[
            {
                "Resource": [
                    "arn:aws:sqs:us-west-2:1111111:Dev-ELK-LogstashBuffer"
                ],
                "Action":[
                    "sqs:SendMessage",
                    "sqs:GetQueueAttributes",
                    "sqs:GetQueueUrl",
                    "sqs:ListQueues",
                    "sqs:ChangeMessageVisibility"
                ],
                "Effect":"Allow"
            }
        ],
        "Version":"2012-10-17"
    }