Thanks for your help and sorry for the long post!
I would like to ensure python dependencies are installed in the instance of Cloud Composer. A developer will need to install a dependency and so they'll submit the code to Bitbucket including these, and a Bitbucket pipeline will manage the installation in the Cloud composer instance when deploying the changes.
To manage dependencies, we (have to) use poetry and, among other things, the deployment also syncs the DAG folder with a bucket, installs poetry, generates a requirements.txt file and attempts to update the PyPI packages of the composer using the gcloud CLI following documentation
This last step, installing dependencies, is what I'm having issues with.
Below just the part of the bitbucket-pipeline.yaml that is relevant below:
- /etc/poetry/bin/poetry export --without-hashes --format requirements.txt --output /tmp/requirements_tmp.txt
- sed "s/\"/'/g" /tmp/requirements_tmp.txt >> /tmp/requirements.txt
- gcloud composer environments update ${COMPOSER_PROJECT_ID}-composer --location ${COMPOSER_LOCATION} --update-pypi-packages-from-file /tmp/requirements.txt
poetry works fine.sed) replaces double quotes for single, thinking that could be the error - however the same error persists whether I add this step or not. I also tried a different combination that adds spaces between library and version but with the same error message.Given the error message (below):
requirements.txt file is generated correctly (apparently). I have 8 errors, exactly for the dependencies I was trying to add. (Documentation).
cat the file to the pipeline to confirm (see below anyway).pip install -r requirements.txt, with the contents of the output generated by poetry, it works fine.poetry not generating a full PEP compliant requirement.txt file ? Is there any (other) transformation to the file that I should be doing ?
| version | |
|---|---|
| composer | composer-2.1.11-airflow-2.4.3 |
| airflow | 2.4.3 |
| python | 3.8.12 |
| poetry | 1.4.2 |
Please note that after running the sed command it would successfully replace double quotes with single quotes.
colorama==0.4.6 ; python_full_version == "3.8.12" and sys_platform == "win32"
exceptiongroup==1.0.1 ; python_full_version == "3.8.12"
iniconfig==2.0.0 ; python_full_version == "3.8.12"
packaging==21.3 ; python_full_version == "3.8.12"
pluggy==1.0.0 ; python_full_version == "3.8.12"
pyparsing==3.0.9 ; python_full_version == "3.8.12"
pytest==7.3.1 ; python_full_version == "3.8.12"
tomli==2.0.1 ; python_full_version == "3.8.12"
gcloud composer environments update ${COMPOSER_PROJECT_ID}-composer --location ${COMPOSER_LOCATION} --update-pypi-packages-from-file /tmp/requirements.txt
ERROR: (gcloud.composer.environments.update) INVALID_ARGUMENT: Found 8 problems:
1) Error validating PyPI package extras and version ==0.4.6 ; python_full_version == '3.8.12' and sys_platform == 'win32'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
2) Error validating PyPI package extras and version ==1.0.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
3) Error validating PyPI package extras and version ==2.0.0 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
4) Error validating PyPI package extras and version ==21.3 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
5) Error validating PyPI package extras and version ==1.0.0 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
6) Error validating PyPI package extras and version ==3.0.9 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
7) Error validating PyPI package extras and version ==7.3.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
8) Error validating PyPI package extras and version ==2.0.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508 (https://www.python.org/dev/peps/pep-0508/).
As per our discussion in the comments, this issue was resolved by editing the script by removing the part after ; ie. python_full_version was removed from the requirement.txt
To automate this, a snippet that edits the generated requirements.txt to only include PyPI <package><version> was written.