I'm behind a company proxy with a self-signed certificate and I want to install tensorstore via pip. pip apparently downloads and runs a Python script bazelisk.py that in turn uses urllib to get more stuff from the Internet. However, this fails with a CERTIFICATE_VERIFY_FAILED error message:
$ pip install --trusted-host=example.com --index-url=http://example.com/pypi/simple
...
Downloading https://releases.bazel.build/6.4.0/release/bazel-6.4.0-linux-arm64...
Traceback (most recent call last):
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 1346, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 1040, in _send_output
self.send(msg)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 980, in send
self.connect()
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/pip-install-ycop_psv/tensorstore_1008eee73d464825b2e191c044b9e306/bazelisk.py", line 492, in <module>
sys.exit(main())
File "/tmp/pip-install-ycop_psv/tensorstore_1008eee73d464825b2e191c044b9e306/bazelisk.py", line 477, in main
bazel_path = get_bazel_path()
File "/tmp/pip-install-ycop_psv/tensorstore_1008eee73d464825b2e191c044b9e306/bazelisk.py", line 470, in get_bazel_path
return download_bazel_into_directory(bazel_version, is_commit, bazel_directory)
File "/tmp/pip-install-ycop_psv/tensorstore_1008eee73d464825b2e191c044b9e306/bazelisk.py", line 304, in download_bazel_into_directory
download(bazel_url, destination_path)
File "/tmp/pip-install-ycop_psv/tensorstore_1008eee73d464825b2e191c044b9e306/bazelisk.py", line 353, in download
with closing(urlopen(request)) as response, open(destination_path, "wb") as file:
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 517, in open
response = self._open(req, data)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 534, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 1389, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/user/anaconda3/envs/PyTorch-1.11.0/lib/python3.9/urllib/request.py", line 1349, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)>
error: command '/home/user/anaconda3/envs/PyTorch-1.11.0/bin/python3.9' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for tensorstore
Failed to build tensorstore
ERROR: Could not build wheels for tensorstore which use PEP 517 and cannot be installed directly
I already know this error message from software such as Huggingface, and managed to solve it in many cases.
I already put the required company certificates to /etc/pki/ca-trust/source/anchors and run update-ca-trust afterwards (Note: I'm on a CentOS-derived distro). By verifying the timestamps and contents of /etc/pki/tls/cert.pem, I made sure the update was successful. curl https://www.google.com works. But pip install still fails.
So, I pip install certifi, and retried. Still fails. certifi actually installs the certificates from requests, which naturally don't include our company's self-signed certificates. So, I replaced certifi's PEM file with a link to above-mentioned /etc/pki/tls/cert.pem:
mv "$(python -m certifi)"{,.bak}
ln -s "/etc/pki/tls/cert.pem" "$(python -m certifi)"
But this changes nothing. pip install tensorstore still fails with above-mentioned error.
Setting REQUESTS_CA_BUNDLE doesn't work here, because requests is not involved.
At this point, I'm don't know further. Any solution presented to me online only revolves around the solutions I already mentioned. Not even a quick-and-dirty verify=False equivalent, PYTHONHTTPSVERIFY, or unverified SSL context could be applied to my current situation because the script run is temporary, downloaded every time I try to install.
So, how could I tackle this problem?
Setting SSL_CERT_FILE successfully managed to guide Python's urllib to the correct PEM file:
$ export SSL_CERT_FILE=/etc/pki/tls/cert.pem
$ pip install --trusted-host=example.com --index-url=http://example.com/pypi/simple tensorstore
But now, another, possibly unrelated problem occurred. SSL certificate problem in Java...