If I had to perform ETL on a huge dataset(say 1Tb) stored in S3 as csv files, Both AWS Glue ETL job and AWS EMR steps can be used. Then how is AWS Glue different from AWS EMR. And which is the better solution in this case.
Most of the differences are already listed so I'll focus more on the use case specific.
When to choose aws glue
- Data size is huge but structured i.e. it is in the table structure and is of known format (CSV, parquet, orc, json).
- Lineage is required, if you need the data lineage graph while developing your etl job prefer developing the etl using glue native libraries.
- The developers don't need to tweak the performance parameters like setting number of executors, per executor memory and so on.
- You don't want the overhead of managing large cluster and pay only for what you use.
When to use EMR
- Data is huge but semi-structured or unstructured where you can't take any benefit from Glue catalog.
- You believe only in the outputs and lineage is not required.
- You need to define more memory per executor depending upon the type of your job and requirement.
- You can manage the cluster easily or if you have so many jobs which can run concurrently on the cluster saving you money.
- In case of structured data, you should use EMR when you want more Hadoop capabilities like hive, presto for further analytics.
So it depends on what your use case is. Both are great service.