Search code examples
amazon-web-servicesaws-cloudformationamazon-athenaaws-glueamazon-quicksight

How to use Glue/Athena/Quicksight with CloudFormation


I have some data in S3, created a schema in the Glue catalog, and then exposed it to QuickSight via Athena. All this works great when I create it by clicking in the console.

I then converted it to the following CloudFormation:

AnalyticsDatabase:
  Type: AWS::Glue::Database
  Properties:
    DatabaseInput: 
      Name: analytics
    CatalogId: !Ref AWS::AccountId
RawAnalysisAnalyticsTable:
  Type: AWS::Glue::Table
  Properties:
    DatabaseName: !Ref AnalyticsDatabase
    CatalogId: !Ref AWS::AccountId
    TableInput:
      Name: analysis_raw
      TableType: EXTERNAL_TABLE
      Parameters:
        classification: json
      StorageDescriptor:
        Columns:
          - {Name: id, Type: string}
          - {Name: treeid, Type: string}
          - {Name: patientid, Type: string}
        Compressed: false
        InputFormat: org.apache.hadoop.mapred.TextInputFormat
        OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
        Location: s3://my-bucket/dynamodb/Analysis/
        NumberOfBuckets: 0
        SerdeInfo:
          Parameters: {paths: 'id,patientid,treeid'}
          SerializationLibrary: org.openx.data.jsonserde.JsonSerDe
        SortColumns: []
        StoredAsSubDirectories: false

However, when I try to pull the CF-created table into QuickSight I get:

Your database generated a SQL exception. This can be caused by query timeouts, resource constraints, unexpected DDL alterations before or during a query, and other database errors. Check your database settings and your query, and try again.

region:             us-east-1
timestamp:          1544113019756
requestId:          5ab8f9a2-f972-11e8-b201-154c30728c75
sourceErrorCode:    0
sourceErrorMessage: [Simba][JDBC](11380) Null pointer exception.
sourceErrorState:   HY000
sourceException:    java.sql.SQLException
sourceType:         ATHENA

Does anyone have any idea what this error means or how I can troubleshoot it? I've compared all the properties of the manually-created table to the CloudFormation-created table, and they seem identical.


Solution

  • Max's answer should be the accepted answer here. Replicated this and only solution that worked was to add the PartitionKeys: [] parameter. I had initially added it as a child or StorageDescription, which didn't work. Has to be added at the TableInput child level as specified in the docs. It is the right answer because none of the other conditions listed here (security, etc) will give the NullPointerException that is referenced in the question.