Search code examples
aws-sam-cli

Running AWS SAM build from within Python script

I'm in the process of migrating entire CloudFormation stacks to Troposphere, including Lambda and Lambda-reliant CFN Custom Resources.

One of my goals is to circumvent the creation of template files altogether, making the Python code the sole "source of truth" (i.e without template files that are created and therefore can be edited, causing config drift).

This requires the ability to:

  1. Pass a file-like object to the SAM builder (instead of a file-name)

  2. Calling the AWS SAM builder from Python and not the CLI

My first naive idea was that I would be able to import a few modules from aws-sam-cli put a wrapper for io.StringIO around it (to hold the template as file-like object) and presto! Then I looked at the source code for sam build and all hope left me:

  • I may not be able to use Docker/containers for building, as I it will map the build environment, including template files.

  • AWS SAM CLI is not designed to have a purely callable set of library functions, similar to boto3. Close, but not quite.

Here is the core of the Python source

with BuildContext(template,
                  base_dir,
                  build_dir,
                  clean=clean,
                  manifest_path=manifest_path,
                  use_container=use_container,
                  parameter_overrides=parameter_overrides,
                  docker_network=docker_network,
                  skip_pull_image=skip_pull_image,
                  mode=mode) as ctx:

    builder = ApplicationBuilder(ctx.function_provider,
                                 ctx.build_dir,
                                 ctx.base_dir,
                                 manifest_path_override=ctx.manifest_path_override,
                                 container_manager=ctx.container_manager,
                                 mode=ctx.mode
                                 )
    try:
        artifacts = builder.build()
        modified_template = builder.update_template(ctx.template_dict,
                                                    ctx.original_template_path,
                                                    artifacts)

        move_template(ctx.original_template_path,
                      ctx.output_template_path,
                      modified_template)

        click.secho("\nBuild Succeeded", fg="green")

        msg = gen_success_msg(os.path.relpath(ctx.build_dir),
                              os.path.relpath(ctx.output_template_path),
                              os.path.abspath(ctx.build_dir) == os.path.abspath(DEFAULT_BUILD_DIR))

        click.secho(msg, fg="yellow")

This relies on a number of imports from a aws-sam-cli internal library with the build focused ones being

from samcli.commands.build.build_context import BuildContext
from samcli.lib.build.app_builder import ApplicationBuilder, BuildError, UnsupportedBuilderLibraryVersionError, ContainerBuildNotSupported
from samcli.lib.build.workflow_config import UnsupportedRuntimeException

It's clear that this means it's not as simple as creating something like a boto3 client and away I go! It looks more like I'd have to fork the whole thing and throw out nearly everything to be left with the build command, context and environment.

Interestingly enough, sam package and sam deploy, according to the docs, are merely aliases for aws cloudformation package and aws cloudformation deploy, meaning those can be used in boto3!

Has somebody possibly already solved this issue? I've googled and searched here, but haven't found anything.

I use PyCharm and the AWS Toolkit which if great for development and debugging and from there I can run SAM builds, but it's "hidden" in the PyCharm plugins - which are written in Kotlin!

My current work-around is to create the CFN templates as temp files and pass them to the CLI commands which are called from Python - an approach I've always disliked.

I may put in a feature request with the aws-sam-cli team and see what they say, unless one of them reads this.

Solution

  • I've managed to launch sam local start-api from a python3 script.

    Firstly, pip3 install aws-sam-cli

    Then the individual command can be imported and run.

    import sys
from samcli.commands.local.start_api.cli import cli
sys.exit(cli())

    ... provided there's a template.yaml in the current directory.

    What I haven't (yet) managed to do is influence the command-line arguments that cli() would receive, so that I could tell it which -t template to use.

    Edit

    Looking at the way aws-sam-cli integration tests work it seems that they actually kick off a process to run the CLI. So they don't actually pass a parameter to the cli() call at all :-(

    For example:

    class TestSamPython36HelloWorldIntegration(InvokeIntegBase):
    template = Path("template.yml")

    def test_invoke_returncode_is_zero(self):
        command_list = self.get_command_list(
            "HelloWorldServerlessFunction", template_path=self.template_path, event_path=self.event_path
        )

        process = Popen(command_list, stdout=PIPE)
        return_code = process.wait()

        self.assertEquals(return_code, 0)

   .... etc

    from https://github.com/awslabs/aws-sam-cli/blob/a83aa9e620ff679ca740496a3f1ff4872b88894a/tests/integration/local/invoke/test_integrations_cli.py

    See also start_api_integ_base.py in the same repo.

    I think on the whole this is to be expected because the whole thing is implemented in terms of the click command-line application framework. Unfortunately.

    See for example http://click.palletsprojects.com/en/7.x/testing/ which says "The CliRunner.invoke() method runs the command line script in isolation ..." -- my emphasis.