How can I fix the circular dependency between my S3 bucket and SQS?

Question

I am trying to write a serverless configuration for my service. A requirement is that the S3 bucket sends notifications to an SQS queue on object create events. However, when I try to deploy my service using serverless deploy, I get this error:

Serverless Error ----------------------------------------

 An error occurred: PolicyS3Bucket - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 4D25CQFZN0R2Q9FG; S3 Extended Request ID: dLfKHJgOnDUcAF3xwN9EgW9LibP3bt7ITj7PyuCXs2qH6Qvmn2iZu7aXYbbUdqptPvgvjwkcWYM=; Proxy: null).

I found this page which (if I understand correctly) explains that I have a circular dependency between my S3 bucket and my SQS queue, and that I must fix this circular dependency in order to be able to successfully deploy my service.

This page explains that I can use Fn::Sub or Fn::Join to fix the circular dependency. Based on this suggestion, I modified my configuration from the original version to a new version as below, using Sub:

cfn.s3.yml (original version)

Resources:
  PolicyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ${self:custom.config.policyBucketName}
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      NotificationConfiguration:
        QueueConfigurations:
          - Event: s3:ObjectCreated:*
            Queue: !GetAtt SQSQueue.Arn
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - BucketKeyEnabled: true
            ServerSideEncryptionByDefault:
              KMSMasterKeyID: !Ref CustomMasterKey
              SSEAlgorithm: aws:kms
      Tags: ${redacted}

cfn.s3.yml (new version, change in bold)

Resources:
  PolicyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ${self:custom.config.policyBucketName}
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      NotificationConfiguration:
        QueueConfigurations:
          - Event: s3:ObjectCreated:*
            Queue: !Sub arn:aws:sqs:${self:provider.region}:${AWS::AccountId}:${self:custom.config.sqsQueueName}
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - BucketKeyEnabled: true
            ServerSideEncryptionByDefault:
              KMSMasterKeyID: !Ref CustomMasterKey
              SSEAlgorithm: aws:kms
      Tags: ${redacted}

My unchanged cfn.sqs.yml

Resources:
  SQSQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: ${self:custom.config.sqsQueueName}

When I tried serverless deploy with the new version, I get the same error.

I also tried @kgiannakakis's suggestion to use DependsOn, but I get the same error when I try that.

How can I fix my serverless configuration so that I can successfully deploy my service?

Answer 1

I found a fix that worked.

I had to update my cfn.sqs.yml to include permissions for S3 buckets to send events to the SQS queue, as below:

Resources:
  SQSQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: ${self:custom.config.sqsQueueName}
  S3EventQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    DependsOn: SQSQueue
    Properties:
      PolicyDocument:
        Id: SQSPolicy
        Statement:
          - Effect: Allow
            Sid: PutS3Events
            Action: SQS:SendMessage
            Resource: !GetAtt SQSQueue.Arn
            Principal:
              Service: s3.amazonaws.com
      Queues:
        - !Ref SQSQueue

As for my cfn.s3.yml, the correct way to reference the queue was

Queue: !GetAtt SQSQueue.Arn

I believe the problem was that AWS checks that the notification will be possible at deployment time, rather than letting your service fail at runtime, as explained in this answer:

A lot of AWS configuration allows you to connect services and they fail at runtime if they don't have permission, however S3 notification configuration does check some destinations for access.

This would mean that, since I hadn't configured my SQS queue to allow notifications from the S3 bucket, AWS noticed this misconfiguration and stopped the deployment with an error.