Using maven to create python deployment package for aws lambda

Question

I am currently working on deploying python package on aws lambda. Using virtualenv and zip tools, I can create a lambda zip file easily to be uploaded to aws and run it. I use the following blog to create the package: https://joarleymoraes.com/hassle-free-python-lambda-deployment/.

However, I need to use a build tool to integrate my code with Jenkins in my company. We use maven for building and I need to follow the same.

I looked at maven-exec-plugin to see how I can follow the steps in the blog to use virtual env and create a zip file. However, I am not able to follow the steps mentioned in the maven tutorial.

Has anyone come across the same problem? If so, how have you solved it? As in, how have you configured your pom.xml to create a deployment package for python aws lambdas?

Quick help will be highly appreciated. Thanks

Answer 1

Let me break this question down into two parts. First, for a python lambda you have to create a zip file. For this I recommend using the Maven Assembly Plugin.

Sample pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.mygroup</groupId>
        <artifactId>parent-pom</artifactId>
        <version>0.1-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
    </parent>
    <artifactId>LambdaInstall</artifactId>
    <name>Python Lambda Installation Package</name>
    <packaging>pom</packaging>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
          <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <executions>
              <execution>
                <id>install-assembly</id>
                <phase>package</phase>
                <goals>
                  <goal>single</goal>
                </goals>
                <configuration>
                  <appendAssemblyId>false</appendAssemblyId>
                  <descriptors>
                    <descriptor>assembly.xml</descriptor>
                  </descriptors>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
    </build>
</project>

Sample assembly.xml:

<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
  <id>lambda-zip</id>
  <formats>
    <format>zip</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <fileSets>
      <fileSet>
        <directory>src/main/python</directory>
        <outputDirectory></outputDirectory>
      </fileSet>
  </fileSets>
</assembly>

If your lambda only uses the standard python libraries, then this is all you need. The complication comes in when you need boto3 or some other library which AWS does not pre-install on its lambda nodes. For this you have to add your libraries into your zip file.

I wanted a way to do it in a way that would work on any developer's PC or even on a Jenkins node so I wrote a small python app which finds the location of the library (under site-packages directory) and copies it into the application target directory. In this example I needed jwt:

import jwt
import distutils
from distutils import dir_util

# This will copy the jwt package into the target directory so it is included in the zip file
jwtPath = jwt.__path__[0]
distutils.dir_util.copy_tree(jwtPath, "./target/jwt")

This python program is invoked by adding the Maven Exec Plugin into my pom.xml:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>python-build</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>exec</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
       <executable>python</executable>
       <arguments>
            <argument>src/main/python/build/copyjwt.py</argument>
       </arguments>    
    </configuration>
</plugin>

Now in order to copy the code into my zip file, I need only add one more file set to the assembly.xml that I showed above:

<fileSet>
    <directory>target/jwt</directory>
    <outputDirectory>jwt</outputDirectory>
</fileSet>