Search code examples
elasticsearchamazon-s3aws-gluedata-migration

Loading data from S3 to Elasticsearch using AWS glue

I have muliple folders in a S3 bucket and each folder contains one JSON lines file.

I want to do two things with this data

  1. Apply some transformations and get tabular data and save it to some database.
  2. save these json objects, as it is to Elasticseach cluster for full text search

I am using AWS glue for this task and I know how to do 1, but, I can't find any resources that talks about getting data from s3 and storing it to elasticsearch using AWS glue.

Is there a way to do this?

Solution

  • If anyone is looking for an answer to this then I used Logstash to load files to Elasticsearch.