If I run the following command from my development box:
$ openssl s_client -connect github.com:443
I get the following last line of output:
Verify return code: 20 (unable to get local issuer certificate)
If I try to do this with requests I get another failed request:
>>> import requests
>>> r = requests.get('https://github.com/', verify=True)
With an exception raised:
SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I can also run the first command with the verify flag and get similar output:
$ openssl s_client -connect github.com:443 -verify 9
...
Verify return code: 27 (certificate not trusted)
Basically this is telling me that there is a problem with the certificates. I can specify a specific certificate with both methods and it will work:
$ openssl s_client -connect github.com:443 -CAfile /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -verify 9
...
Verify return code: 0 (ok)
and:
>>> r = requests.get('https://github.com/', verify='/etc/ssl/certs/DigiCert...pem')
<Response [200]>
So, to my question, what exactly is wrong here? Shouldn't requests/openssl already know where to find valid certs?
Other Info:
Also, I know passing verify=False to the requests.get method will work too, but I do want to verify.
EDIT
I've confirmed that, as @Heikki Toivonen indicated in an answer, specifying the -CAfile flag for the version of openssl that I'm running works.
$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
...
Verify return code: 0 (ok)
So there is nothing wrong with the version of openssl that I'm running, and there is nothing wrong with the default cacert.pem file that requests provides.
Now that I know openssl is meant to work that way, that the CAfile or the place to find certs has to be specified, I'm more concerned about getting requests to work.
If I run:
>>> r = requests.get('https://github.com/', verify='path to cacert.pem file')
I'm still getting the same error as before. I even tried downloading the cacert.pem file from http://curl.haxx.se/ca and it still didn't work. requests only seems to work (on this specific machine) if I specify a specific vendor cert file.
A side note: On my local machine everything is working as expected. There are several difference between the two machines though. I so far haven't been able to determine what the specific difference is that causes this issue.
If I run the following command from my development box:
$ openssl s_client -connect github.com:443I get the following last line of output:
Verify return code: 20 (unable to get local issuer certificate)
You are missing DigiCert High Assurance EV CA-1 as a root of trust:
$ openssl s_client -connect github.com:443
CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
...
Start Time: 1393392088
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Download DigiCert High Assurance EV CA-1 from DigiCert Trusted Root Authority Certificates:
$ wget https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
--2014-02-26 00:27:50-- https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
Resolving www.digicert.com (www.digicert.com)... 64.78.193.234
...
Convert the DER encoded certifcate to PEM:
$ openssl x509 -in DigiCertHighAssuranceEVCA-1.crt -inform DER -out DigiCertHighAssuranceEVCA-1.pem -outform PEM
Then, use it with OpenSSL via the -CAfile:
$ openssl s_client -CAfile DigiCertHighAssuranceEVCA-1.pem -connect github.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify return:1
depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 5157550, street = 548 4th Street, postalCode = 94107, C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
verify return:1
---
Certificate chain
0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHOjCCBiKgAwIBAgIQBH++LkveAITSyvjj7P5wWDANBgkqhkiG9w0BAQUFADBp
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSgwJgYDVQQDEx9EaWdpQ2VydCBIaWdoIEFzc3VyYW5j
ZSBFViBDQS0xMB4XDTEzMDYxMDAwMDAwMFoXDTE1MDkwMjEyMDAwMFowgfAxHTAb
BgNVBA8MFFByaXZhdGUgT3JnYW5pemF0aW9uMRMwEQYLKwYBBAGCNzwCAQMTAlVT
MRkwFwYLKwYBBAGCNzwCAQITCERlbGF3YXJlMRAwDgYDVQQFEwc1MTU3NTUwMRcw
FQYDVQQJEw41NDggNHRoIFN0cmVldDEOMAwGA1UEERMFOTQxMDcxCzAJBgNVBAYT
AlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xEzARBgNVBAMTCmdpdGh1Yi5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDt04nDXXByCfMzTxpydNm2WpVQ
u2hhn/f7Hxnh2gQxrxV8Gn/5c68d5UMrVgkARWlK6MRb38J3UlEZW9Er2TllNqAy
GRxBc/sysj2fmOyCWws3ZDkstxCDcs3w6iRL+tmULsOFFTmpOvaI2vQniaaVT4Si
N058JXg6yYNtAheVeH1HqFWD7hPIGRqzPPFf/jsC4YX7EWarCV2fTEPwxyReKXIo
ztR1aE8kcimuOSj8341PTYNzdAxvEZun3WLe/+LrF+b/DL/ALTE71lmi8t2HSkh7
bTMRFE00nzI49sgZnfG2PcVG71ELisYz7UhhxB0XG718tmfpOc+lUoAK9OrNAgMB
AAGjggNUMIIDUDAfBgNVHSMEGDAWgBRMWMsl8EFPUvQoyIFDm6aooOaS5TAdBgNV
HQ4EFgQUh9GPGW7kh29TjHeRB1Dfo79VRyAwJQYDVR0RBB4wHIIKZ2l0aHViLmNv
bYIOd3d3LmdpdGh1Yi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjBjBgNVHR8EXDBaMCugKaAnhiVodHRwOi8vY3JsMy5k
aWdpY2VydC5jb20vZXZjYTEtZzIuY3JsMCugKaAnhiVodHRwOi8vY3JsNC5kaWdp
Y2VydC5jb20vZXZjYTEtZzIuY3JsMIIBxAYDVR0gBIIBuzCCAbcwggGzBglghkgB
hv1sAgEwggGkMDoGCCsGAQUFBwIBFi5odHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9z
c2wtY3BzLXJlcG9zaXRvcnkuaHRtMIIBZAYIKwYBBQUHAgIwggFWHoIBUgBBAG4A
eQAgAHUAcwBlACAAbwBmACAAdABoAGkAcwAgAEMAZQByAHQAaQBmAGkAYwBhAHQA
ZQAgAGMAbwBuAHMAdABpAHQAdQB0AGUAcwAgAGEAYwBjAGUAcAB0AGEAbgBjAGUA
IABvAGYAIAB0AGgAZQAgAEQAaQBnAGkAQwBlAHIAdAAgAEMAUAAvAEMAUABTACAA
YQBuAGQAIAB0AGgAZQAgAFIAZQBsAHkAaQBuAGcAIABQAGEAcgB0AHkAIABBAGcA
cgBlAGUAbQBlAG4AdAAgAHcAaABpAGMAaAAgAGwAaQBtAGkAdAAgAGwAaQBhAGIA
aQBsAGkAdAB5ACAAYQBuAGQAIABhAHIAZQAgAGkAbgBjAG8AcgBwAG8AcgBhAHQA
ZQBkACAAaABlAHIAZQBpAG4AIABiAHkAIAByAGUAZgBlAHIAZQBuAGMAZQAuMH0G
CCsGAQUFBwEBBHEwbzAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu
Y29tMEcGCCsGAQUFBzAChjtodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln
aUNlcnRIaWdoQXNzdXJhbmNlRVZDQS0xLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqG
SIb3DQEBBQUAA4IBAQBfFW1nwzrVo94WnEUzJtU9yRZ0NMqHSBsUkG31q0eGufW4
4wFFZWjuqRJ1n3Ym7xF8fTjP3fdKGQnxIHKSsE0nuuh/XbQX5DpBJknHdGFoLwY8
xZ9JPI57vgvzLo8+fwHyZp3Vm/o5IYLEQViSo+nlOSUQ8YAVqu6KcsP/e612UiqS
+UMBmgdx9KPDDzZy4MJZC2hbfUoXj9A54mJN8cuEOPyw3c3yKOcq/h48KzVguQXi
SdJbwfqNIbQ9oJM+YzDjzS62+TCtNSNWzWbwABZCmuQxK0oEOSbTmbhxUF7rND3/
+mx9u8cY//7uAxLWYS5gIZlCbxcf0lkiKSHJB319
-----END CERTIFICATE-----
subject=/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
---
No client certificate CA names sent
---
SSL handshake has read 4139 bytes and written 446 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 59D2883BBCE8E81E63E5551FAE7D1ACC00C49A9473C1618237BBBB0DD9016B8D
Session-ID-ctx:
Master-Key: B6D2763FF29E77C67AD83296946A4D44CDBA4F37ED6F20BC27602F1B1A2D137FACDEAC862C11279C01095594F9776F79
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1393392673
Timeout : 300 (sec)
Verify return code: 0 (ok)
Shouldn't requests/openssl already know where to find valid certs?
No. OpenSSL trusts nothing by default. Its a polar opposite of a browser's model, where nearly everything is trusted by default.
$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
...
>>> r = requests.get('https://github.com/', verify='path to cacert.pem file')
Why would you trust hundreds of CAs and subordinate CAs (re: cacert.pem) when you know the one CA that is certifying the public key for the site? Trust the one required root and nothing more: DigiCert High Assurance EV CA-1.
Trusting everything - as in the browser's model - is what allowed Comodo Hacker to spoof certificates for Gmail, Hotmail, Yahoo, etc when the Diginotar root was compromised.