Specifying Glue::Crawler with JdbcTargets on CloudFormation

Question

I'm trying to set up AWS Glue to read from a RDS Postgres using CloudFormation. In order to do that I need to create a crawler using the JdbcTarget option. (Or do I not?)

  Records:
    Type: 'AWS::Glue::Crawler'
    Properties:
      DatabaseName: transact
      Targets:
        JdbcTargets:
          - Path: "jdbc:postgresql://host:5432/database"
      Role: !Ref ETLAgent

But creating the stack on CloudFormation wil fail with:

CREATE_FAILED | AWS::Glue::Crawler | Records | Connection name cannot be equal to null or empty. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException;

Even though the docs say:

ConnectionName

The name of the connection to use for the JDBC target.

Required: No

What is the correct AWS Glue setup using CloudFormation that will allow me to read from RDS?

Answer 1

You're really missing the ConnectionName property, which should carry the name of connection resource which you're missing. The Path property you're setting is used to select the schemas/tables to crawl (dbname/%/% to include all). Consult CloudFormation docs on Crawler JDBCTarget for details.

Your template should look something like

  MyDbConnection:
    Type: "AWS::Glue::Connection"
    Properties:
      CatalogId: !Ref 'AWS::AccountId'
      ConnectionInput:
        Description: "JDBC Connection to my RDS DB"
        PhysicalConnectionRequirements:
          AvailabilityZone: "eu-central-1a"
          SecurityGroupIdList:
           - my-sec-group-id
          SubnetId: my-subnet-id
        ConnectionType: "JDBC"
        ConnectionProperties:
          "JDBC_CONNECTION_URL": "jdbc:postgresql://host:5432/database"
          "USERNAME": "my-db-username"
          "PASSWORD": "my-password"
  Records:
    Type: 'AWS::Glue::Crawler'
    Properties:
      DatabaseName: transact
      Targets:
        JdbcTargets:
          - ConnectionName: !Ref MyDbConnection
            Path: "database/%/%"
      Role: !Ref ETLAgent