AWS Cloudwatch alarms in Datadog

Question

Looking through Datadog AWS integration documentation I found mention that AWS alarms can be streamed into Datadog. It's stated that you can choose two different methods how to send AWS CloudWatch alarms to the Datadog Event Stream right here in the Alarm collection section. But there is no further explanation on how to do that or what should be set upped to do that. Moreover, trying to google something like "Datadog aws alarm polling" gives you vague descriptions of some other functionalities but not the AWS CloudWatch alarms.

My question is is it even possible ?

What I tried so far is set upped DataDog Lambda Forwarder that sends CloudWatch logs (metrics and alarms I suppose too?) to DD. I gave permission to that lambda. I created some AWS metric filter and AWS alarm to trigger when specific event occurs. I run some lambda code to throw exception and trigger CloudWatch alarm to change its status.

I clearly see lambda logs in DD, but I can't find anything related to my alarms in DD events. And I suppose it's not a problem with DD-AWS integration because we use it in big organization and it was configured long before I joined the company. What am I doing wrong ?

Cloudformation script below (I removed some parts, so it's not working as is)

Resources:
  DatadogForwarderLambda:
    Type: AWS::Lambda::Function
    Properties:
      Description: Pushes logs, metrics and traces from AWS to Datadog.
      Role: !GetAtt "DatadogForwarderLambdaRole.Arn"
      Handler: lambda_function.lambda_handler
      Code:
        S3Bucket: config-sandbox
        S3Key: 'aws-dd-forwarder-3.38.0.zip'
      MemorySize: 1024
      Runtime: python3.7
      Timeout: 120
      Tags:
        - Key: "dd_forwarder_version"
          Value:  3.38.0
      Environment:
        Variables:
          DD_ENHANCED_METRICS: "false"
          DD_API_KEY_SECRET_ARN: 
            Ref: DdApiKeySecret
          DD_S3_BUCKET_NAME: config-sandbox
          DD_SITE: datadoghq.com
          DD_: datadoghq.com
          DD_TAGS_CACHE_TTL_SECONDS: 300
          DD_FETCH_LAMBDA_TAGS: true
          DD_USE_TCP: false
          DD_NO_SSL: false
          REDACT_IP: false
          REDACT_EMAIL: false
          DD_USE_PRIVATE_LINK: false
          DD_USE_VPC: false
      ReservedConcurrentExecutions: 100


  DatadogReadonlyPolicy:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: !Sub "DatadogReadonlyPolicy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - 'cloudwatch:Get*'
              - 'cloudwatch:List*'
              - 'cloudwatch:DescribeAlarmHistory'
              - 'cloudtrail:LookupEvents'
              - 'ec2:Describe'
              - 's3:GetObject'
              - 's3:PutObject'
              - 's3:DeleteObject'
              - 's3:ListBucket'
              - 'lambda:List*'
              - 'tag:GetResources'
              - 'tag:GetTagKeys'
              - 'tag:GetTagValues'
              - 'support:*'
            Resource: !GetAtt DatadogForwarderLambda.Arn
          - Effect: Allow
            Action:
              - secretsmanager:GetSecretValue
            Resource:
              - Ref: DdApiKeySecret
      Roles: 
        - !Ref DatadogForwarderLambdaRole
       

  DatadogForwarderLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
              AWS:
                - Fn::Sub:
                  - "arn:aws:iam::${AccountId}:role/human-role/some-role-name"
                  - { AccountId: !Ref 'AWS::AccountId' }
            Action:
              - sts:AssumeRole    
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
      Path: /
      PermissionsBoundary:
        Fn::Join:
              - ''
              - - 'arn:aws:iam::'
                - Ref: AWS::AccountId
                - ':policy/some-organisation-permission-boundary'
      RoleName:               
        Fn::Sub:
        - 'a${AIID}-dd-forwarder-lambda-${StackID}'
        - { StackID: !Select [4, !Split ["-", !Ref 'AWS::StackId']],
            AIID: !Ref AIID }


  IncomingQueueHasMessagesExceptionAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Incoming queue has unprocessed messages, new processing round can't be started
      AlarmName: !Sub "IncomingQueueHasMessagesExceptionAlarm"
      ComparisonOperator: GreaterThanThreshold
      Threshold: 0 # no messages are allowed in queue if new round started
      EvaluationPeriods: 1
      Period: 10  
      Namespace: dev-logs
      MetricName: QueueHasMessagesException
      Statistic: Sum   
      TreatMissingData: missing


  IncomingQueueHasMessagesExceptionMetricFilter: 
    Type: AWS::Logs::MetricFilter
    Properties: 
      LogGroupName: 
        !Sub '/aws/lambda/${SomeLambdaName}'
      FilterPattern: "QueueHasMessagesException"
      MetricTransformations: 
        - 
          MetricNamespace: dev-logs
          MetricName: QueueHasMessagesException
          MetricValue: 1
  

Answer 1

Eventually I found that my AWS account was not completely integrated into DD.