I cannot find a way to pip install the following Python modules without compatibility issues (from a requirements.txt file, on Red Hat Enterprise Linux release 8.2):
tensorflow_gpu
tensorflow_addons
tensorflow_io
pandas
pillow
matplotlib
I tried to specify versions like
tensorflow_gpu
tensorflow_addons==2.6.0
tensorflow_io
pandas
pillow
matplotlib
however, the installation always fails.
I tried the above mentioned within a virtual environment. During my latest attempts, I removed the site packages to avoid further compatibility issues:
~$ mkdir ./venv
~$ python3 -m venv ./venv
~$ source ./venv/bin/activate
(venv) ~$ cd ./venv
(venv) ./venv$ pip3.8 freeze | xargs pip3.8 uninstall -y
(venv) ./venv$ pip3.8 install -r requirements.txt
After the installation finished with problems, I cannot import tensorflow and tensorflow_addons like this:
(venv) ./venv$ python3
>>> import tensorflow as tf
>>> import tensorflow_addons as tfa
because of the following error (full output further down):
tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists
Is there a place where I can find matching versions or is the issue not resolvable by simply specifying certain versions of the modules?
Full module import error output:
2022-02-09 06:29:07.367472: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow_addons/__init__.py", line 21, in <module>
from tensorflow_addons import activations
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow_addons/activations/__init__.py", line 17, in <module>
from tensorflow_addons.activations.gelu import gelu
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow_addons/activations/gelu.py", line 19, in <module>
from tensorflow_addons.utils.types import TensorLike
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow_addons/utils/types.py", line 24, in <module>
from keras.engine import keras_tensor
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/__init__.py", line 25, in <module>
from keras import models
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/models.py", line 20, in <module>
from keras import metrics as metrics_module
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/metrics.py", line 26, in <module>
from keras import activations
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/activations.py", line 20, in <module>
from keras.layers import advanced_activations
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/layers/__init__.py", line 23, in <module>
from keras.engine.input_layer import Input
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/engine/input_layer.py", line 21, in <module>
from keras.engine import base_layer
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/engine/base_layer.py", line 43, in <module>
from keras.mixed_precision import loss_scale_optimizer
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/mixed_precision/loss_scale_optimizer.py", line 18, in <module>
from keras import optimizers
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/optimizers.py", line 26, in <module>
from keras.optimizer_v2 import adadelta as adadelta_v2
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/optimizer_v2/adadelta.py", line 22, in <module>
from keras.optimizer_v2 import optimizer_v2
File "/opt/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages/keras/optimizer_v2/optimizer_v2.py", line 36, in <module>
keras_optimizers_gauge = tf.__internal__.monitoring.BoolGauge(
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow/python/eager/monitoring.py", line 360, in __init__
super(BoolGauge, self).__init__('BoolGauge', _bool_gauge_methods,
File "/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages/tensorflow/python/eager/monitoring.py", line 135, in __init__
self._metric = self._metric_methods[self._label_length].create(*args)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.
During module installation This is the output during installation of the modules in requirements.txt:
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Requirement already satisfied: tensorflow==2.6.0 in /pfs/data5/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (2.6.0)
Collecting tensorflow_gpu==2.6.0
Using cached tensorflow_gpu-2.6.0-cp38-cp38-manylinux2010_x86_64.whl (458.4 MB)
Requirement already satisfied: tensorflow_addons in /pfs/data5/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (0.15.0)
Requirement already satisfied: tensorflow_io in /pfs/data5/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (0.21.0)
Requirement already satisfied: pandas in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (1.4.0)
Requirement already satisfied: pillow in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from -r requirements.txt (line 6)) (9.0.0)
Requirement already satisfied: matplotlib in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from -r requirements.txt (line 7)) (3.5.1)
Requirement already satisfied: wheel~=0.35 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (0.37.1)
Collecting absl-py~=0.10
Using cached absl_py-0.15.0-py3-none-any.whl (132 kB)
Requirement already satisfied: google-pasta~=0.2 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (0.2.0)
Collecting six~=1.15.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting flatbuffers~=1.12.0
Using cached flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: tensorflow-estimator~=2.6 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (2.7.0)
Collecting typing-extensions~=3.7.4
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Requirement already satisfied: keras~=2.6 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (2.7.0)
Requirement already satisfied: termcolor~=1.1.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (1.1.0)
Collecting h5py~=3.1.0
Using cached h5py-3.1.0-cp38-cp38-manylinux1_x86_64.whl (4.4 MB)
Collecting numpy~=1.19.2
Using cached numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
Requirement already satisfied: gast==0.4.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (0.4.0)
Requirement already satisfied: grpcio<2.0,>=1.37.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (1.43.0)
Requirement already satisfied: opt-einsum~=3.3.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (3.3.0)
Requirement already satisfied: tensorboard~=2.6 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (2.8.0)
Collecting clang~=5.0
Using cached clang-5.0-py3-none-any.whl
Collecting wrapt~=1.12.1
Using cached wrapt-1.12.1-cp38-cp38-linux_x86_64.whl
Requirement already satisfied: protobuf>=3.9.2 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (3.19.4)
Requirement already satisfied: astunparse~=1.6.3 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorflow==2.6.0->-r requirements.txt (line 1)) (1.6.3)
Requirement already satisfied: typeguard>=2.7 in /pfs/data5/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages (from tensorflow_addons->-r requirements.txt (line 3)) (2.13.3)
Requirement already satisfied: tensorflow-io-gcs-filesystem==0.21.0 in /pfs/data5/home/kit/ifgg/mp3890/.local/lib/python3.8/site-packages (from tensorflow_io->-r requirements.txt (line 4)) (0.21.0)
Requirement already satisfied: pytz>=2020.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from pandas->-r requirements.txt (line 5)) (2021.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from pandas->-r requirements.txt (line 5)) (2.8.2)
Requirement already satisfied: fonttools>=4.22.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from matplotlib->-r requirements.txt (line 7)) (4.29.0)
Requirement already satisfied: pyparsing>=2.2.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from matplotlib->-r requirements.txt (line 7)) (3.0.7)
Requirement already satisfied: cycler>=0.10 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from matplotlib->-r requirements.txt (line 7)) (0.11.0)
Requirement already satisfied: packaging>=20.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from matplotlib->-r requirements.txt (line 7)) (21.3)
Requirement already satisfied: kiwisolver>=1.0.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from matplotlib->-r requirements.txt (line 7)) (1.3.2)
Requirement already satisfied: setuptools>=41.0.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (41.6.0)
Requirement already satisfied: requests<3,>=2.21.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (2.27.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (1.8.1)
Requirement already satisfied: markdown>=2.6.8 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (3.3.6)
Requirement already satisfied: google-auth<3,>=1.6.3 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (2.5.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (0.6.1)
Requirement already satisfied: werkzeug>=0.11.15 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (2.0.2)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (0.4.6)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (4.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (4.10.1)
Requirement already satisfied: idna<4,>=2.5 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (3.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (1.26.8)
Requirement already satisfied: certifi>=2017.4.17 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (2.0.11)
Requirement already satisfied: zipp>=0.5 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (3.7.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->-r requirements.txt (line 1)) (3.2.0)
Installing collected packages: wrapt, typing-extensions, flatbuffers, clang, six, numpy, h5py, absl-py, tensorflow_gpu
Attempting uninstall: wrapt
Found existing installation: wrapt 1.13.3
Not uninstalling wrapt at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'wrapt'. No files were found to uninstall.
Attempting uninstall: typing-extensions
Found existing installation: typing_extensions 4.0.1
Not uninstalling typing-extensions at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'typing_extensions'. No files were found to uninstall.
Attempting uninstall: flatbuffers
Found existing installation: flatbuffers 2.0
Not uninstalling flatbuffers at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'flatbuffers'. No files were found to uninstall.
Attempting uninstall: six
Found existing installation: six 1.16.0
Not uninstalling six at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'six'. No files were found to uninstall.
Attempting uninstall: numpy
Found existing installation: numpy 1.22.1
Not uninstalling numpy at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'numpy'. No files were found to uninstall.
Attempting uninstall: h5py
Found existing installation: h5py 3.6.0
Not uninstalling h5py at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'h5py'. No files were found to uninstall.
Attempting uninstall: absl-py
Found existing installation: absl-py 1.0.0
Not uninstalling absl-py at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'absl-py'. No files were found to uninstall.
Attempting uninstall: tensorflow_gpu
Found existing installation: tensorflow-gpu 2.7.0
Not uninstalling tensorflow-gpu at /pfs/data5/software_uc2/bwhpc/common/jupyter/tensorflow/2022-01-11/lib/python3.8/site-packages, outside environment /pfs/data5/home/kit/ifgg/mp3890/ven0
Can't uninstall 'tensorflow-gpu'. No files were found to uninstall.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
bokeh 2.4.2 requires typing-extensions>=3.10.0, but you have typing-extensions 3.7.4.3 which is incompatible.
black 22.1.0 requires typing-extensions>=3.10.0.0; python_version < "3.10", but you have typing-extensions 3.7.4.3 which is incompatible.
Successfully installed absl-py-0.15.0 clang-5.0 flatbuffers-1.12 h5py-3.1.0 numpy-1.19.5 six-1.15.0 tensorflow_gpu-2.6.0 typing-extensions-3.7.4.3 wrapt-1.12.1
WARNING: You are using pip version 22.0.2; however, version 22.0.3 is available.
You should consider upgrading via the '/pfs/data5/home/kit/ifgg/mp3890/ven0/bin/python3 -m pip install --upgrade pip' command.
Following the suggestion to invoke Python with -m pip (as python3 -m pip install -r requirements.txt) led to the same output (except for that message).
The problem was caused by jupyter/tensorflow being loaded in the background. The following solved the issue:
module purge
module load devel/cuda
/usr/bin/python3.8 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
Many thanks to Samuel Braun from our cluster support team