Search code examples
pythonsslaws-cloudformationurllib3

How to download server SSL certificate with urllib3?

The title really is the question — how do I get urllib3 to download the SSL cert from the remote server when trying to make an HTTPS connection?

Background Over in ServerVault land, I'm trying to get to the bottom of a problem with AWS autoscaling groups failing to be able to download phpMyAdmin as part of their init process. As part of trying to diagnose that, I'm now over in StackOverflow land, working with python scripts. Here's where I am:

#! /usr/bin/python3
import cfnbootstrap
from cfnbootstrap.packages import requests
from requests import utils
from requests.utils import DEFAULT_CA_BUNDLE_PATH
from requests.packages import urllib3

conn = urllib3.connection_from_url("https://dev.mysql.com/", retries=False)
conn.cert_reqs = 'CERT_REQUIRED'
conn.ca_certs = DEFAULT_CA_BUNDLE_PATH
response = conn.request("GET", "/get/")
conn.close()
print(response.status)

That works fine:

$ ./urltest.py 
404

However, if I change it to be

conn = urllib3.connection_from_url("https://www.phpmyadmin.net", retries=False)
response = conn.request("GET", "/downloads/")

I run into my problem:

$ ./urltest.py 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 762, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connection.py", line 238, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/util/ssl_.py", line 265, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib64/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib64/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)

It seems very unlikely that the certificate has expired, as if I go check manually by using a web browser, all is good with the world; however, I'd like to see exactly what certificate urllib3 is seeing as having expired. How do I do this?

NB: I'm not calling certify.where() directly, as I'm not sure what shananigans AWS is performing in the background, but since their scripts are auto-installed as part of booting the instance, there's not much I can do about it, so I'm trying to use their internal processes.

Solution

  • The problem is the expiration of the Let's Encrypt DST Root CA X3 that happened on Sept.30 -- https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/. The requests package doesn't trust the ISRG Root X1 cert and is trying to verify it with the expired cert.

    Download the ISRG X1 PEM from https://letsencrypt.org/certificates/ and add it to the cfnbootstrap library's cacert.pem file.

    cat isgrootx1.pem >> /path/to/your/python/env/lib/python3.9/site-packages/cfnbootstrap/packages/requests/cacert.pem

    You can find the location of your cacert.pem with the following:

    from cfnbootstrap.packages.requests.certs import where
print(where())

    Also... I would change up your code to make sure you're actually using the cfn packaged requests module rather than the standalone requests. Otherwise, you probably will have to update it in both spots.

    from cfnbootstrap.packages.requests.utils import DEFAULT_CA_BUNDLE_PATH
from cfnbootstrap.packages.requests.packages import urllib3

conn = urllib3.connection_from_url("https://www.phpmyadmin.net/", retries=False)
conn.cert_reqs = 'CERT_REQUIRED'
conn.ca_certs = DEFAULT_CA_BUNDLE_PATH
response = conn.request("GET", "/get/")
conn.close()
print(response.status)

    If you do want to update the standalone requests and the one packaged with cfnbootstrap, use the following to find and update both cacert paths:

    import requests
import cfnbootstrap.packages.requests

print(requests.certs.where())
print(cfnbootstrap.packages.requests.certs.where())