Have searched the AWS Glue documents, but could not find the pricing details for AWS Glue worker types G.1X and G.2X. Can someone please explain if there is no cost difference between Standard, G.1X & G.2X?
All I can see the Glue pricing section is "You are billed $0.44 per DPU-hour in increments of 1 second, rounded up to the nearest second. Glue Spark jobs using Glue version 2.0 have a 1-minute minimum billing duration. ". Is this irrespective of the worker type?
Standard type - 16 GB memory, 4 vCPUs of compute capacity, and 50 GB of attached EBS storage (2 Executors)
G.1X - 16 GB memory, 4 vCPUs, and 64 GB of attached EBS storage (1 Executor)
G.2X - Twice that of G.1X (https://aws.amazon.com/blogs/big-data/best-practices-to-scale-apache-spark-jobs-and-partition-data-with-aws-glue/) which means,
G.2X - 32 GB memory, 8vCPUs, 128 GB of EBS !!
Appreciate any inputs on this.
Yuva
As you can read up here:
When you are using G1.X / G.2X you are allocating an amount of worker. Those map to DPU.
For the G.1X worker type, each worker maps to 1 DPU
and
For the G.2X worker type, each worker maps to 2 DPU
That means that G.2X is twice as costly as G.1X. If you are using Standard, you are allocating a specific amount of DPU directly. If you are using Glue 2.0 I would advise you to use either G.1X or G.2X, depending on your use case.