I am trying to reproduce this tutorial to run a Flex Template on Dataflow.
Few minutes after submitting the job, it fails with this error:
Output from execution of subprocess: b'Collecting apache-beam[gcp]==2.26.0
Downloading apache-beam-2.26.0.zip (2.4 MB)
Saved /tmp/dataflow-requirements-cache/apache-beam-2.26.0.zip
Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1
Downloading avro-python3-1.9.2.1.tar.gz (37 kB)
Saved /tmp/dataflow-requirements-cache/avro-python3-1.9.2.1.tar.gz
Collecting crcmod<2.0,>=1.7
Downloading crcmod-1.7.tar.gz (89 kB)
Saved /tmp/dataflow-requirements-cache/crcmod-1.7.tar.gz
Collecting dill<0.3.2,>=0.3.1.1
Downloading dill-0.3.1.1.tar.gz (151 kB)
Saved /tmp/dataflow-requirements-cache/dill-0.3.1.1.tar.gz
Collecting fastavro<2,>=0.21.4
Downloading fastavro-1.2.2.tar.gz (663 kB)
Saved /tmp/dataflow-requirements-cache/fastavro-1.2.2.tar.gz
Collecting future<1.0.0,>=0.18.2
Downloading future-0.18.2.tar.gz (829 kB)
Saved /tmp/dataflow-requirements-cache/future-0.18.2.tar.gz
Collecting grpcio<2,>=1.29.0
Downloading grpcio-1.34.0.tar.gz (21.0 MB)
Saved /tmp/dataflow-requirements-cache/grpcio-1.34.0.tar.gz
Collecting hdfs<3.0.0,>=2.1.0
Downloading hdfs-2.5.8.tar.gz (41 kB)
Saved /tmp/dataflow-requirements-cache/hdfs-2.5.8.tar.gz
Collecting httplib2<0.18.0,>=0.8
Downloading httplib2-0.17.4.tar.gz (256 kB)
Saved /tmp/dataflow-requirements-cache/httplib2-0.17.4.tar.gz
Collecting mock<3.0.0,>=1.0.1
Downloading mock-2.0.0.tar.gz (73 kB)
Saved /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting numpy<2,>=1.14.3
Downloading numpy-1.19.4.zip (7.3 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Saved /tmp/dataflow-requirements-cache/numpy-1.19.4.zip
Collecting pymongo<4.0.0,>=3.8.0
Downloading pymongo-3.11.2.tar.gz (770 kB)
Saved /tmp/dataflow-requirements-cache/pymongo-3.11.2.tar.gz
Collecting oauth2client<5,>=2.0.1
Downloading oauth2client-4.1.3.tar.gz (155 kB)
Saved /tmp/dataflow-requirements-cache/oauth2client-4.1.3.tar.gz
Collecting protobuf<4,>=3.12.2
Downloading protobuf-3.14.0.tar.gz (227 kB)
Saved /tmp/dataflow-requirements-cache/protobuf-3.14.0.tar.gz
Collecting pyarrow<0.18.0,>=0.15.1
Downloading pyarrow-0.17.1.tar.gz (2.6 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_p9la_sd
cwd: /tmp/pip-download-uramyun1/pyarrow
Complete output (42 lines):
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
main()
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 254, in run_setup
self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 145, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 605, in <module>
url='https://arrow.apache.org/'
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.7/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 424, in __init__
k: v for k, v in attrs.items()
File "/usr/local/lib/python3.7/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 695, in finalize_options
ep(self)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 702, in _finalize_setup_keywords
ep.load()(self, ep.name, value)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/integration.py", line 26, in version_keyword
dist.metadata.version = _get_version(config)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 173, in _get_version
parsed_version = _do_parse(config)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 119, in _do_parse
parse_result = _call_entrypoint_fn(config.absolute_root, config, config.parse)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 54, in _call_entrypoint_fn
return fn(root)
File "setup.py", line 539, in parse_git
return parse(root, **kwargs)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/git.py", line 95, in parse
require_command("git")
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/utils.py", line 151, in require_command
raise EnvironmentError("%r was not found" % name)
OSError: 'git' was not found
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_p9la_sd Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Any one else have the same issue ?
If it can help, here is the Dockerfile of the image:
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
# https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
COPY streaming_beam.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/streaming_beam.py"
RUN pip install -U -r ./requirements.txt
Looks like you need to install git in your dockerfile. Add this line:
RUN apt-get install git