Search code examples
amazon-web-servicesaws-cliaws-glue

Get number of partitions in AWS Glue for specific range

I want to list all the partitions for a given table and get a count of it, but aws glue get-partitions --database-name ... returns detailed information about each partitions which is not very helpful in this case.

Let's say my table is partitioned by input_data_date and country I want to know how many partitions I have for a given day.

I can do something with this

aws glue get-partitions --database-name MYDB --table-name MYTABLE --expression "input_data_date = '2021-07-09' "

But it needs some scripting I was looking for a better and cleaner way just by AWS CLI or ....

Solution

  • The AWS CLI uses JMESPATH, which has a length() function. Therefore, you can use:

    aws glue get-partitions --database-name xx --table-name xx --query 'length(Partitions[])'

    That will return the total number of partitions.

    If you want to do something more specific ("how many partitions I have for a given day"), you'd probably need to use a better SDK (eg Python with boto3) to process the information.