so I am trying to get partitions from my Glue table. But I need only a certain set of partitions that pass the condition that is applied to the partition's value.
I did try the other way of getting all the partitions and then filtering them out, but it is making a lot of API calls.
I checked the boto3 documentation but I don't think that I could apply a filter expression on the partition's value. Nor could I find any example of a filter expression being applied.
The value of partition is as follows
[1, '202210051', 1, 123]
The get_partitions will only pull partitions that matches the expression. I have partitions of below format
dt=2022-09-07/partition_1=08
dt=2022-09-08/partition_1=07
dt=2022-09-12/partition_1=06
dt=2022-09-16/partition_1=05
If I want to filter for only partition then the expression will be of below format:
expression="dt = '2022-09-07' AND partition_1 = 08"
response = client.get_partitions(
DatabaseName='test_glue',
TableName='testahana_ahana_test_test_ql_8vuvs70w9h',
Expression=expression,
MaxResults=123
)
Similarly depending on your requirement you can pass the conditional expression to get_partitions