I am using Spark to write files to S3 in ORC format. Also using Athena to query this data.
I am using the following partition keys:
s3://bucket/company=1123/date=20190207
Once I execute the Glue crawler to run on the bucket everything works as expected except the types of the partitions keys.
The Crawler configures them in the catalog as String type instead of int
Is there a configuration to define the default type of the partition keys ?
I know it can be changed manually later and set the Crawler config to Add new columns only.
Glue crawlers always treat partition keys as type string and unfortunately there is no configuration option available to change this behavior.