Search code examples
bashamazon-web-servicesaws-cliamazon-sqsamazon-sns

AWS SQS not receiving SNS messages


I created a SNS topic that publishes all the information coming out of Cloudformation via the cli. However, when I check the queue, it is not receiving any of the SNS messages. I verified the SNS is working by subscribing my email to it, so the issue seems to be in the connection between the queue and the SNS. However, I cannot find any problems with my syntax. I, as far as I know, have followed amazon's documentation precisely.

Bash:

#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"

#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`

#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`

#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}
aws sns subscribe --topic-arn ${SNS_ARN} --protocol email-json --notification-endpoint ${EMAIL}

#Create the stack which kicks everything else off-
aws cloudformation create-stack $REGIONTEXT $ITYPETEXT --capabilities CAPABILITY_IAM --template-url https://${BUCKETNAME}.s3.amazonaws.com/${TEMPLATE} --notification-arns ${SNS_ARN} --stack-name $NAME --parameters ParameterKey=SNSARN,ParameterValue=${SNS_ARN} ParameterKey=Bucket,ParameterValue=${BUCKETNAME} ${PARAMTEXT} ${EXTRAARGS}

Solution

  • Thank you to Mark B for his answer. It provided the start to getting this working. However, in order to make a policy document work via the CLI there are a few quirks that aren't covered in the docs.

    1. There are all sorts of errors trying to pass json directly to the --attributes flag in aws sqs set-queue-attributes command. For some reason it requires the modifying json to be in a .json document referenced by the cli.
    2. In the .json file provided to the cli, all of the double quotes inside the "Policy" value (nested json) must be escaped (i.e. { \"Statement\": \"HelloWorld\" }). If this is not followed, it will validation errors. I ended up needing to use the ascii escape characters in order properly format the output (\x5C).
    3. The json file must be referenced by using file://local-location in the --attributes flag. It throws errors if this is not followed.

    See the following elements I used for reference:

    load_sqs.sh:

    SQS_POLICY=
    sqs-policy()
    {
    #First param is the queue arn, second param is the topic arn
    SQS_POLICY=`printf '{ "Policy": "{\x5C\"Version\x5C\":\x5C\"2012-10-17\x5C\",\x5C\"Statement\x5C\":[{\x5C\"Sid\x5C\":\x5C\"CloudformationLambdaSQSPolicy\x5C\",\x5C\"Effect\x5C\":\x5C\"Allow\x5C\",\x5C\"Principal\x5C\":\x5C\"*\x5C\",\x5C\"Action\x5C\":\x5C\"sqs:SendMessage\x5C\",\x5C\"Resource\x5C\":\x5C\"%s\x5C\",\x5C\"Condition\x5C\":{\x5C\"ArnEquals\x5C\":{\x5C\"aws:SourceArn\x5C\":\x5C\"%s\x5C\"}}}]}" }' "$1" "$2"`
    `echo $SQS_POLICY > $PWD/sqs-policy.json`
    }
    
    #SNS parameters
    SNS_NAME="${NAME}_SNS"
    SQS_NAME="${NAME}_SQS"
    
    #Create SNS topic to send cloudformation notifications to
    SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`
    
    #Create SQS to send SNS to (holding SNS messages for lambda -^ up)
    SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
    SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`
    
    #Add necessary SQS <--> SNS permissions
    sqs-policy ${SQS_ARN} ${SNS_ARN}
    `aws sqs set-queue-attributes --queue-url ${SQS_URL} --attributes file://sqs-policy.json`
    
    #subscribe the queue to the notifications
    aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}
    

    sqs-policy.json:

    { "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"CloudformationLambdaSQSPolicy\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"ResourceARN\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"SourceARN\"}}}]}" }