I have an AWS CloudFormation JSON template as follows:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "CloudFormation template to import an existing S3 bucket and setup automation stack.",
"Parameters": {
"InputS3BucketName": {
"Description": "The name of your existing S3 bucket, to which your MDF log files are uploaded.",
"Type": "String",
"AllowedPattern": "^[a-z0-9]{1}[a-z0-9\\.\\-]{0,62}[a-z0-9]{1}$",
"ConstraintDescription": "Bucket name can only contain lowercase letters, numbers, hyphens, and periods. It must start and end with a lowercase letter or number."
},
"UniqueID": {
"Description": "A unique ID (max 10 chars) to distinguish your created resources.",
"Type": "String",
"MaxLength": 10,
"AllowedPattern": "^[a-z0-9]*$",
"ConstraintDescription": "The unique ID can only contain lowercase letters/numbers and should be a maximum of 10 characters."
}
},
"Resources": {
"S3LimitedAccessUser": {
"Type": "AWS::IAM::User",
"Properties": {
"UserName": {
"Fn::Sub": "S3LimitedAccessUser-${UniqueID}"
}
}
},
"S3LimitedAccessPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": {
"Fn::Sub": "S3LimitedAccessPolicy-${UniqueID}"
},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}/*"
},
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}"
}
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}-parquet/*"
},
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}-parquet"
}
]
}
]
},
"Users": [
{
"Ref": "S3LimitedAccessUser"
}
]
}
},
"S3LimitedAccessUserAccessKey": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "S3LimitedAccessUser"
}
}
},
"LambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
],
"Policies": [
{
"PolicyName": {
"Fn::Sub": "S3LimitedAccessPolicy-${UniqueID}"
},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}/*"
},
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}"
}
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
{
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}-parquet/*"
}
]
}
]
}
}
]
}
},
"S3InvokePermission": {
"Type": "AWS::Lambda::Permission",
"DependsOn": "LambdaFunction",
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Ref": "LambdaFunction"
},
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"SourceArn": {
"Fn::Sub": "arn:aws:s3:::${InputS3BucketName}"
}
}
},
"LambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": {
"Fn::Sub": "mdf-to-parquet-lambda-function-${UniqueID}"
},
"Handler": "lambda_function.lambda_handler",
"Timeout": 120,
"MemorySize": 1024,
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
},
"Runtime": "python3.11",
"Code": {
"S3Bucket": "css-electronics-resources",
"S3Key": "lambda/mdf-to-parquet-lambda-function-v1.5.0.zip"
}
}
},
"InputS3Bucket": {
"Type": "AWS::S3::Bucket",
"DependsOn": "S3InvokePermission",
"DeletionPolicy": "Retain",
"Properties": {
"BucketName": {
"Ref": "InputS3BucketName"
},
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": [
"LambdaFunction",
"Arn"
]
},
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": ".MF4"
}
]
}
}
},
{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": [
"LambdaFunction",
"Arn"
]
},
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": ".MFC"
}
]
}
}
},
{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": [
"LambdaFunction",
"Arn"
]
},
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": ".MFE"
}
]
}
}
},
{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": [
"LambdaFunction",
"Arn"
]
},
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": ".MFM"
}
]
}
}
}
]
}
}
},
"OutputS3Bucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Fn::Join": [
"",
[
{
"Ref": "InputS3BucketName"
},
"-parquet"
]
]
},
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"BlockPublicPolicy": true,
"IgnorePublicAcls": true,
"RestrictPublicBuckets": true
}
}
}
},
"Outputs": {
"01": {
"Description": "S3 Access Key for copying data from local disk (or S3 input bucket) to output bucket",
"Value": {
"Ref": "S3LimitedAccessUserAccessKey"
}
},
"02": {
"Description": "S3 Secret Key for copying data from local disk (or S3 input bucket) to output bucket",
"Value": {
"Fn::GetAtt": [
"S3LimitedAccessUserAccessKey",
"SecretAccessKey"
]
}
},
"03": {
"Description": "Input S3 bucket name (where MDF files are stored)",
"Value": {
"Fn::Sub": "${InputS3BucketName}"
}
},
"04": {
"Description": "Output S3 bucket name (where Parquet files are stored)",
"Value": {
"Fn::Sub": "${InputS3BucketName}-parquet"
}
},
"05": {
"Description": "Default Region",
"Value": {
"Ref": "AWS::Region"
}
}
}
}
The template is to be applied as a change set on an imported 'input' S3 bucket. The stack works fine when used on an eu-central-1 input bucket when I am in the region eu-central-1 in my AWS console.
However, if I switch to using another region, e.g. us-east-2, it won't work and I get an error on the Lambda function requesting to shift to eu-central-1.
I assume the problem is that the Lambda function zip is hosted on an AWS S3 bucket within the eu-central-1 region. This is done to allow users across different regions to fetch the Lambda function zip via their own AWS CFT deployments using this stack. My expectation would be that it would not matter where the Lambda function was hosted regionally for this purpose, but it seems that it does.
Is there a simple way to resolve this - or do I really have to create duplicate S3 buckets in every region of relevance in order to host the Lambda function zip in a way so that a user can leverage this stack template regardless of what AWS S3 region they use?
do I really have to create duplicate S3 buckets in every region of relevance in order to host the Lambda function zip
Yes, as the documentation says:
S3Bucket:
An Amazon S3 bucket in the same AWS Region as your function. The bucket can be in a different AWS account.
What you can do is use S3 Cross-region replication so you don't have to manually copy the zip files on all regional buckets.