Using poetry to install dependencies in Azure Functions pipeline

Question

Goal
Deploy a python project including a function app using poetry and the pyproject.toml file. The specified modules should be accessible by the python files.

Background
My project is hosted in a Azure DevOps repository. With an Azure YAML Pipeline it is "built" with poetry install, creating a sub-folder .venv with all dependencies, and bundled into a zip file. The archive is stored as an artifact and pulled onto the VM in the deploy step, where I extract it, so that the .venv folder is present again.
My expectation was that activating the venv with poetry env use .venv/bin/python.exe or source .venv/bin/activate was sufficient. Both steps work and do activate the venv, but it does not stick and modules are not accessible in python scripts. Locally, running poetry install is sufficient and works very well.

Challenges
A) poetry installs dependencies in a different location than what is accessible
Usually the solution is to use pip install with the argument --target to place dependencies in the folder ./.python_packages/lib/site-packages. poetry does not have a functionality like this, unfortunately.

B) activating virtual environments is only valid in a single steps of pipelines and gets lost afterwards
With source venv/bin/activate activating the poetry environment works fine, as a combined step with activation and pip list demonstrates:

- bash: |
    source .venv/bin/activate
    pip list

--> lists all dependecies that were installed using poetry install

- bash: |
    source .venv/bin/activate
- bash: |
    pip list

--> lists only dependecies of the Azure base virtual environment (i.e. no pandas)

C) as consequence, running the function app leads to Exception: ModuleNotFoundError: No module named 'pandas'

Question
How can I use poetry in Azure pipelines, so that installed dependencies are accessible by python scripts?

Answer 1

Solving the issue involved two parts.

1. Making sure the dependencies get installed to the correct VM during the Azure pipeline.

2. Getting the dependencies installed at the correct location, so that Python scripts can actually access them.

For 1., I ended up not using a build and dependency step in my pipeline.yml, but to only use a single step, with a deployment step. That way only one VM is initialized and I only have to install poetry once.

For 2., exporting to a requirements.txt and installing with pip --target was the only way to get it to run correctly in my situation. Note that there is a poetry extension that aims to solve this (https://github.com/python-poetry/poetry-plugin-bundle).

Here is the pipeline with all relevant parts:

trigger:
- main

variables:
  pythonVersion: 3.10
  azureSubscription: <YOUR_SUBSCRIPTION_CODE>
  functionAppName: <YOUR_FUNCTION_APP_NAME>
  vmImageName: 'ubuntu-latest'
  workingDirectory: '$(System.DefaultWorkingDirectory)'
jobs:
- deployment: Deploy
  environment: <YOUR_ENV_NAME>
  pool:
    vmImage: $(vmImageName)
  strategy:
    runOnce:
      deploy:
        steps:
        - checkout: self
        - task: UsePythonVersion@0
          inputs:
            versionSpec: $(pythonVersion)
        - bash: |
            curl -sSL https://install.python-poetry.org | python -
            export PATH=$PATH:$HOME/.poetry/bin
          displayName: 'Install poetry'
        - bash: |
            poetry lock --no-update
            poetry install --no-root --without dev,local
          displayName: 'Update lock file and install dependencies'
        - bash: |
            poetry env use /home/vsts/work/1/s/.venv/bin/python
          displayName: 'Activate virtual environment'
        - bash: |
            poetry export -f requirements.txt --output requirements.txt
            pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
          displayName: 'Export poetry dependencies and install'
        - task: ArchiveFiles@2
          inputs:
            rootFolderOrFile: '$(workingDirectory)'
            includeRootFolder: false
            archiveType: zip
            archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
            replaceExistingArchive: true
          displayName: 'Archive files'
        - publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
          artifact: drop
          displayName: 'Publish artifact'
        - task: AzureFunctionApp@2
          inputs:
            azureSubscription: '$(azureSubscription)'
            appType: functionAppLinux
            appName: $(functionAppName)
            package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
            deploymentMethod: 'auto'
          displayName: 'Deploy Azure function app'

Feel free to ask any questions you have about details and particular steps.