How to use zappa in gitlab CI/CD to deploy app to AWS Lambda?

Question

I am trying to deploy a flask application on aws lambda via zappa through gitlab CI. Since inline editing isn't possible via gitlab CI, I generated the zappa_settings.json file on my remote computer and I am trying to use this to do zappa deploy dev.

My zappa_settings.json file:

{
    "dev": {
        "app_function": "main.app",
        "aws_region": "eu-central-1",
        "profile_name": "default",
        "project_name": "prices-service-",
        "runtime": "python3.7",
        "s3_bucket": -MY_BUCKET_NAME-
    }
}

My .gitlab-ci.yml file:

image: ubuntu:18.04

stages:
  - deploy

before_script:
  - apt-get -y update
  - apt-get -y install python3-pip python3.7 zip
  - python3.7 -m pip install --upgrade pip
  - python3.7 -V
  - pip3.7 install virtualenv zappa

deploy_job:
  stage: deploy
  script:
    - mv requirements.txt ~
    - mv zappa_settings.json ~
    - mkdir ~/forlambda
    - cd ~/forlambda
    - virtualenv -p python3 venv
    - source venv/bin/activate
    - pip3.7 install -r ~/requirements.txt -t ~/forlambda/venv/lib/python3.7/site-packages/
    - zappa deploy dev

The CI file, upon running, gives me the following error:

Any suggestions are appreciated

Answer 1

zappa_settings.json is commited to the repo and not created on the fly. What is created on the fly is AWS credentials file. Values required are being read from Gitlab env vars set in the web UI of the project.

zappa_settings.json

{
    "prod": {
        "lambda_handler": "main.handler",
        "aws_region": "eu-central-1",
        "profile_name": "default",
        "project_name": "dummy-name",
        "s3_bucket": "dummy-name",
        "aws_environment_variables": {
            "STAGE": "prod",
            "PROJECT": "dummy-name"
        }
    },
    "dev": {
        "extends": "prod",
        "debug": true,
        "aws_environment_variables": {
            "STAGE": "dev",
            "PROJECT": "dummy-name"
        }
    }
}

.gitlab-ci.yml

image:
  python:3.6

stages:
  - test
  - deploy

variables:
  AWS_DEFAULT_REGION: "eu-central-1"
  # variables set in gitlab's web gui:
  #   AWS_ACCESS_KEY_ID
  #   AWS_SECRET_ACCESS_KEY

before_script:
  # adding pip cache
  - export PIP_CACHE_DIR="/home/gitlabci/cache/pip-cache"

.zappa_virtualenv_setup_template: &zappa_virtualenv_setup
  # `before_script` should not be overriden in the job that uses this template
  before_script:
    # creating virtualenv because zappa MUST have it and activating it
    - pip install virtualenv
    - virtualenv ~/zappa
    - source ~/zappa/bin/activate

    # installing requirements in virtualenv
    - pip install -r requirements.txt

test code:
  stage: test
  before_script:
    # installing testing requirements
    - pip install -r requirements_testing.txt
  script:
    - py.test

test package:
  <<: *zappa_virtualenv_setup
  variables:
    ZAPPA_STAGE: prod
  stage: test
  script:
    - zappa package $ZAPPA_STAGE

deploy to production:
  <<: *zappa_virtualenv_setup
  variables:
    ZAPPA_STAGE: prod
  stage: deploy
  environment:
    name: production
  script:
    # creating aws credentials file
    - mkdir -p ~/.aws
    - echo "[default]" >> ~/.aws/credentials
    - echo "aws_access_key_id = "$AWS_ACCESS_KEY_ID >> ~/.aws/credentials
    - echo "aws_secret_access_key = "$AWS_SECRET_ACCESS_KEY >> ~/.aws/credentials

    # try to update, if the command fails (probably not even deployed) do the initial deploy
    - zappa update $ZAPPA_STAGE || zappa deploy $ZAPPA_STAGE
  after_script:
    - rm ~/.aws/credentials
  only:
    - master

I haven't used zappa in a while, but I remember that a lot of errors that were caused by bad AWS credentials, but zappa reporting something else.