Search code examples
aws-cloudformationconfluent-schema-registryaws-msk

Provide hostname to Schema Registry instances running within ECS


I am building a redundant Schema Registry hosted in Amazon for our MSK Kafka Cluster by using an ECS cluster.

The SchemaRegistry TaskDefinition needs to define a hostname which is unique to each Task when running.

SchemaRegistryTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref SchemaRegistryTaskName
      RequiresCompatibilities: [ EC2 ]
      NetworkMode: bridge
      Cpu: !Ref CPUReservation
      Memory: !Ref MemoryReservation
      Volumes: []
      ContainerDefinitions:
        - Name: !Ref SchemaRegistryTaskName
          Image: !Ref SchemaRegistryTaskImage
          Essential: true
          PortMappings:
            - ContainerPort: !Ref SchemaRegistryPort
              HostPort: 0 # Randomly assigned port from the ephemeral port range.
          Environment:
            - Name: AWS_DEFAULT_REGION
              Value: !Ref AWS::Region
            - Name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
              Value: !Ref MskBrokerUrls
            - Name: SCHEMA_REGISTRY_HOST_NAME
              Value: $HOSTNAME
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref 'CloudwatchLogsGroup'
              awslogs-region: !Ref 'AWS::Region'

NB: Using $Hostname works when running the docker container directly in an EC2 instance via the cli because shell substitutes in the fully qualified hostname which is unique; but I am stumped trying to figure out how to make this work within ECS & CloudFormation.


Solution

  • In the end I went with a custom Command and EntryPoint on the TaskDefinition ContainerDefinitions; using the metadata endpoint suggested by @Apolozeus:

    EntryPoint: ["/bin/bash"]
    Command: ["-c","(export SCHEMA_REGISTRY_HOST_NAME=$(wget -qO- 169.254.169.254/latest/meta-data/local-ipv4);/etc/confluent/docker/run)"]
    

    This ensures the environment variable for SCHEMA_REGISTRY_HOST_NAME exists on the container and correctly maps to ipv4 of the EC2 instance the container is running on.

    This is preferable to me, because we don't control the Docker container being run (it is publicly available), and I don't want to wrap it with a Docker container that we then have to maintain.