Downloading webpage requiring SSL certificate authentication in Python

I'm trying to scrape an endpoint that requires an SSL certificate for authentication. When I try accessing the site in the browser, a window comes up where I select the certificate to be sent with the request. When downloading the webpage, I've tried doing:

import urllib
urllib.request.urlopen("https://mywebpage.com/some/endpoint", cafile = "C:/Users/Me/path/to/my/cert.p12")

but I got the following error:

ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4149)

I thought this might be because I was sending a PKCS12 instead of a PEM file, so I did the following:

openssl pkcs12 -in "C:/Users/Me/path/to/my/cert.p12" -out "C:/Users/Me/path/to/my/cert.pem"

I provided the same value for the PKCS12 passphrase and the PEM password. I then tried doing this:

import urllib
urllib.request.urlopen("https://mywebpage.com/some/endpoint", cafile = "C:/Users/Me/path/to/my/cert.pem")

However, this returns the following SSL error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

When I do ssl.SSLContext().load_verify_locations(cafile = "C:/Users/Me/path/to/my/cert.pem") I get no errors and the certificate information seems to load.

I then decided to try getting around the issue and tried searching for the specific SSL error. I came across this SO question. Since I'm using Windows, my version of OpenSSL is likely configured to be different from what Python is expecting, so I tried doing the following:

import ssl
import certifi
from urllib import request
request.urlopen("https://mywebpage.com/some/endpoint", 
    context = ssl.create_default_context(cafile = certifi.where()))

but this returned the following SSL error:

ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

So, what am I doing wrong here? It appears that the server was unable to verify the SSL certificate I sent with the request but, when I check it, it appears to match. How can I make this request work?

Solution

CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

This error message is unrelated to the site requiring a client certificate. Instead it is about the client (your program) not being able to validate the server certificate.

While nothing about the server certificate is known it might be caused by using the expected client certificate as cafile argument. But cafile is for specifying the trusted root CA for server validation, not the client certificate. To specify this you have to create a SSL context:

from urllib.request import urlopen
import ssl
ctx = ssl.create_default_context()
ctx.load_verify_locations('ca-which-signed-the-server-certificate.pem')
ctx.load_cert_chain('my-client-cert.pem','my-client-key.pem')
urlopen('https://needs-mtls.example.com', context=ctx)