Search code examples
pythonjsonelasticsearchamazon-sqs

How to configure Elasticsearch to pull messages from Amazon SQS


In my current scenario , i am having ElasticSearch on a ubuntu 14.04 machine and a amazon SQS queue holding some ".json" messages in the queue. I want to pull the messages from amazon SQS via ES so that it can use the .json messages to index it.

What should i be looking for, how exactly should i be proceeding? I am looking for something in lines of python's boto code.

NOTE: I am not interested in using logstash for this


Solution

  • Amazon SQS provides an API to receive messages: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/ReceiveMessage.html

    ElasticSearch provides an API to index messages: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index-doc.html

    I am not sure if ES can automatically receive messages from SQS, but one feasible solution is a simple broker between SQS and ES to receive and index.