Search code examples
pythongoogle-cloud-platformgoogle-cloud-dataflowapache-beamdataflow

Dataflow fails due to ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)

Since the beginning of this year our python dataflow jobs result in an error on worker startup:

Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."

My requirements.txt is

ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0

I am using the following Dockerfile to create a Flex-Template:

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}

# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
#   https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*

ADD utils utils
COPY requirements.txt .
COPY main.py .

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"

RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt

Has anyone a solution for the issue? I did try to set a specific setuptools version (51.1.1) as suggested in the following post leading to the same error message. Has anyone faced the same issue before

Solution

  • The issue was due a conflict in the dataclasses-json (The exact reason I couldn't find out). After removing it from the requirements.txt the image can successfully be buildt:

    ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8