AWS CloudWatch LogGroup never sending logs to Lambda despite SubscriptionFilter

Question

Objective: Set up a Cloudformation stack that takes a log group name as a parameter, and whenever new logs show up in that log group, they get sent to a Lambda function for processing, then to Kinesis Firehose, which then sends the log files to a bucket called foobarbaz.

The problem: The Lambda function is never getting invoked (the Lambda's CloudWatch logs show that it never gets triggered even after new data appears in the log group). The invocation should happen automatically thanks to the SubscriptionFilter resource I set up. I'm not seeing any errors. Whatever is happening appears to be failing silently.

Note: the FilterPattern on the SubscriptionFilter has been set to an empty string. My intention with this is to send ALL logs from the log group to the Lambda function.

Here's my Cloudformation template:

Parameters:
  LogGroupName:
    Type: String
    Description: The name of the log group who's logs we want to send to send to Lambda->Kinesis->S3

  AuditTrailPrefix:
    Type: String
    Description: Log files will be sent to the Logging account S3 bucket with this prefix in the bucket path

Resources:  
  AuditTrailFunctionPermissions:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref AuditTrailFunction
      Principal: logs.amazonaws.com
      SourceAccount: !Ref AWS::AccountId

  AuditTrailFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuditTrailFunctionRole.Arn
      Code:
        ZipFile: >
          // do some stuff with the data and PUT it to KinesisFirehose
          // removed for brevity
      Runtime: nodejs8.10
      Timeout: 30

  AuditTrailFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Effect: Allow
                Resource: !Sub arn:aws:firehose:${AWS::Region}:${AWS::AccountId}:deliverystream/${AuditTrailDeliveryStream}
            Version: '2012-10-17'
          PolicyName: root

  AuditTrailSubscription:
    Type: AWS::Logs::SubscriptionFilter
    DependsOn: AuditTrailFunctionPermissions
    Properties:
      DestinationArn: !GetAtt AuditTrailFunction.Arn
      FilterPattern: ''
      LogGroupName: !Ref LogGroupName

  AuditTrailDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamType: DirectPut
      S3DestinationConfiguration:
        BucketARN: arn:aws:s3:::foobarbaz
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 50
        CompressionFormat: GZIP
        Prefix: !Ref AuditTrailPrefix
        RoleARN: !GetAtt DeliveryRole.Arn

  DeliveryRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Effect: Allow
          Principal:
            Service: firehose.amazonaws.com
          Action: sts:AssumeRole

      Policies:
        - PolicyName: firehose_delivery_policy
          PolicyDocument:
            Statement:
              Effect: Allow
              Action:
                - s3:AbortMultipartUpload
                - s3:GetBucketLocation
                - s3:GetObject
                - s3:ListBucket
                - s3:ListBucketMultipartUploads
                - s3:PutObject
              Resource:
                - arn:aws:s3:::foobarbaz
                - arn:aws:s3:::foobarbaz/${AuditTrailPrefix}*

Answer 1

I can't see anything wrong, but here are some tips to troubleshoot:

1. The Lambda function will only get invoked when new logs get uploaded to your log group. It won't get invoked for data that was already in the log group before you set up the subscription filter.

2. If #1 is not the case (i.e. you have new data getting uploaded), go to CloudWatch -> Metrics and search for the log group name. You should find 4 metrics related to your subscription filter: ForwardedBytes, ForwardedLogEvents, DeliveryErrors, DeliveryThrottling. See this for the description. If the DeliveryErrors or DeliveryThrottling metrics are > 0, then there's a problem.

3. The most likely problem with DeliveryErrors is a permissions issue. I don't see anything wrong with yours, but that's what I would double check first.

4. You can debug the subscription setup manually with the AWS CLI. (See this.) This can help you find out what piece of the setup might be causing the problem.