Search code examples
boto3amazon-athenaamazon-kinesis-firehose

Put Map data via Firehose to Athena

I have a column in Athena with Map type. I have defined the schema in Glue as Map .

I have defined a firehose stream that refers to the glue schema and converts it to parquet format. However, I am getting the following error upon sending data

The schema is invalid. Error parsing the schema: Error: type expected at the position 0 of 'MAP <STRING, INT>' but 'MAP ' is found.

I am using boto3 firehose client to submit data.

Here is what I have tried

records=[{'Data': '{"id": "aac4da11-d166-42bd-83db-c670f5b5676d", "createdutc": "2019-06-27 10:25:33","tokens": {"black": 1, "spot": 1, "joint": 1, "leg": 1, "due": 1, "itchinghow": 1, "reduce": 1}'}]

firehoseClient.put_record_batch(
    DeliveryStreamName='conversations-analysis',
    Records=records)
Solution

  • Apparently, I was able to solve it, mention everything in your schema in small letter and you should be able to save map data. For e.g map instead of MAP