How to bundle Python Lambdas with shared files and different dependencies?

Question

I've got some apparently conflicting concerns when deploying Lambda functions using CDK:

1. I want all the dependencies for the whole project to be installed in a single virtualenv in my local copy of the code, so that the IDE can index everything in one go. This is done with a project-level pyproject.toml and poetry.lock.
2. I want only the minimal dependencies to be installed for each Lambda function. This is done by using Poetry "extras" for each Lambda function, and installing only the relevant extras when bundling each Lambda function.
3. I want to share code between Lambdas, by putting shared files into their common parent directory.

The way I do this currently is clunky:

bundling_options = BundlingOptions(
    image=aws_lambda.Runtime.PYTHON_3_8.bundling_docker_image,
    command=["backend/bundle.bash", directory],
)
return aws_lambda.Code.from_asset(path=".", bundling=bundling_options)

bundle.bash

1. uses poetry export to convert the relevant entries from Poetry to pip requirements.txt format and installs using pip install --target=/asset-output (because Poetry doesn't support installing into a specific directory), then
2. copies the files from directory and its parent directory (the shared files) to /asset-output.

I have to use path="." above because the Docker container needs to read pyproject.toml and poetry.lock from the root of the project. This seems to be causing another issue, where any change to any file in the entire repository causes a redeployment of every Lambda function. This is the main reason I'm looking for alternatives.

Is there a better way to share files between Lambda functions in CDK, where the Lambda functions have different dependencies?

Some bad options:

• I don't think I can use PythonFunction, because it assumes that the requirements.txt file is in the root directory (which is shared between Lambda functions).
• I don't want to duplicate the list of packages, so putting all packages in the root configuration and duplicating the relevant packages in requirements.txt files for each Lambda function is out.
• I don't want to do any horrible code rewriting at bundling time, such as moving all the shared code into subdirectories and changing imports from them.
• A workaround for the specific issue with redeploying every time was to specify the asset_hash in from_asset. Unfortunately, this adds considerable brittleness and complexity (hashing every possibly relevant file) rather than simplifying.
• Manually doing bundling outside of Docker. This would also be semi-complex (about the same as right now, by the looks) and would be more brittle than bundling inside Docker. I'd also have to hash the source files, not the resulting files, because Pip installs are in general not reproducible (I see plenty of differences between two installs of the same package to different directories).

Answer 1

Coming from the Node.js world (NodejsFunction) where dependencies are simply excluded from the bundle if they're not used, I'm unsatisfied with my solution but here it is:

I have a pyproject.toml and poetry.lock file at the root of my repository where I define all of my dependencies for all lambdas. My virtual env uses these dependencies. Then I manually add requirements.txt files in my lambdas/ folder with only the dependencies I need for that lambda ensuring the version matches what I have in pyproject.toml. Then I use PythonFunction throughout my CDK app. See screenshot below of folder setup.

Regarding needing common code shared between lambdas, I added a common/ folder adjacent to lambdas/. Then, I'd mount the common code into the bundling docker container upon build. Example:

fn = lambda_py.PythonFunction(
  self,
  "TransformDataFn",
  bundling=lambda_py.BundlingOptions(
    volumes=DockerVolume(
      container_path="/asset-input/common",
      host_path=path.join(dir_path, "../../common"),
    )
  ),
  ...
)