How to run alembic migrations inside of an AWS Lambda?

Question

I created a CircleCI pipeline that builds an AWS infrastructure using Terraform.

The problem now is how to run the Alembic migrations.

So, how to run alembic migrations inside of an AWS Lambda?

Answer 1

I tried multiple ways until I found a solution. I hope it can help someone. Here is how I started creating an Alembic Lambda layer in CircleCI:

mkdir profile_alembic_lambda_layer
pip install alembic -t profile_alembic_lambda_layer/

Here is my main Terraform code

data "archive_file" "lambda_alembic_migrations_archive" {
  type        = "zip"
  source_dir  = "lambda_alembic_migrations"
  output_path = "lambda_alembic_migrations.zip"
}

data "archive_file" "profile_alembic_lambda_layer_archive" {
  type        = "zip"
  source_dir  = "profile_alembic_lambda_layer"
  output_path = "profile_alembic_lambda_layer.zip"
}

resource "aws_lambda_layer_version" "profile_alembic_lambda_layer" {
  filename   = data.archive_file.profile_alembic_lambda_layer_archive.output_path
  layer_name = "profile_alembic_lambda_layer"
  compatible_runtimes = ["python3.8"]  # Specify the runtime(s) your layer is compatible with
}

resource "aws_lambda_function" "alembic_migrations_lambda" {
  filename      = data.archive_file.lambda_alembic_migrations_archive.output_path
  function_name = "alembic_migrations_lambda_handler"
  role          = aws_iam_role.lambda_execution_role.arn
  handler       = "src.app.lambda_postgresql.lambdas.alembic_migration.alembic_migration_lambda_handler"
  runtime       = var.aws_lambda_python_runtime
  source_code_hash = data.archive_file.lambda_alembic_migrations_archive.output_base64sha256
  depends_on = [
    data.archive_file.lambda_alembic_migrations_archive,
    aws_cloudwatch_log_group.lambda_alembic_migrations_log_group,
  ]
  timeout = 900
  layers = [
              aws_lambda_layer_version.profile_lambda_layer.arn,
              aws_lambda_layer_version.profile_alembic_lambda_layer.arn
           ]

  vpc_config {
    subnet_ids         = var.subnet_ids
    security_group_ids = var.lambda_security_group_ids
  }
  environment {
    variables = {
      DATABASE_HOST = var.postgres_db_host
      DATABASE_PORT = var.postgres_db_port
      DATABASE_NAME = var.postgres_db_name
      DATABASE_USER = var.postgres_db_user
      DATABASE_PASSWORD = var.postgres_db_user_password
      LD_LIBRARY_PATH  = "/var/task"  
    }
  }
}

Here is the Lambda function that I used:

import sys


sys.path.append('/opt')


import alembic.config


def alembic_migration_lambda_handler(event, context):
    """ Create Company Lambda Function Handler """
    print("Start")
    
    alembicArgs = [
        '--raiseerr',
        'upgrade', 'head',
    ]
    alembic.config.main(argv=alembicArgs)

Thanks @mimo for the comment. That was part of the solution!