Simple get/post request blocked in python 3 but not in python 2

Question

I'm working on a simple web scraper in python 3 but when I send a get or a post request, the response is 403. In python 2 works fine though. I'm using the same version of requests libraries in both versions. I have also tried with Verify=False/True but the difference in both versions remains.

• Python version 3.7.4 with OpenSSL 1.1.1c 28 May 2019
• Python version 2.7.16 with OpenSSL 1.0.2q 20 Nov 2018
• requests==2.22.0
• certifi==2019.9.11

from requests import get

url = 'https://www.gamestop.com/'
header = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0',
    'DNT': '1',
    'Upgrade-Insecure-Requests': '1',
    'Connection': 'keep-alive',
    'Host': 'www.gamestop.com'
}
res = get(url, headers=header, verify=False).status_code
print(res)
# 403 when using python 3.7.4
# 200 when using python 2.7.16

Edit by @blhsing:

The list below keeps track of which specific Python versions work and which versions fail according to the comments. So far successes and failures have been consistent for each specific Python version across platforms.

Feel free to edit this section of the question with your own results along with the specific Python versions used to produce the results.

2.7.14 works (blhsing)
2.7.16 works (repl.it)
3.6.5 works (blhsing)
3.6.8 fails (Reinderien and blhsing)
3.7.3 works (wim and blhsing)
3.7.4 fails (repl.it and blhsing)
3.8.0 fails (OP)

Demo on repl.it: Python 2.7.16 and Python 3.7.4

Answer 1

This is the exception thrown by urlib3:

/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning,

According to the latest release notes, section 1.25.5 (2019-09-19):

Add mitigation for BPO-37428 affecting Python <3.7.4 and OpenSSL 1.1.1+ which caused certificate verification to be enabled when using cert_reqs=CERT_NONE. (Issue #1682)

You can follow the issue on Github, it has been closed.

TLDR

User @sethmlarson on Github found this bug on urllib3:

create_urllib3_context():

    # Enable post-handshake authentication for TLS 1.3, see GH #1634. PHA is
    # necessary for conditional client cert authentication with TLS 1.3.
    # The attribute is None for OpenSSL <= 1.1.0 or does not exist in older
    # versions of Python.
    if getattr(context, "post_handshake_auth", None) is not None:
        context.post_handshake_auth = True

setting this value to True will enable verification of server certificates, instead of being disabled.