Login to website using http.client

Question

I am trying to login to a website using http.client in Python using the following code:

import urllib.parse
import http.client
payload = urllib.parse.urlencode({"username": "USERNAME-HERE",
                                 "password": "PASSWORD-HERE",
                                 "redirect": "index.php",
                                 "sid": "",
                                 "login": "Login"})
conn = http.client.HTTPConnection("osu.ppy.sh:80")
conn.request("POST", "/forum/ucp.php?mode=login", payload)
response = conn.getresponse()
data = response.read()

# print the HTML after the request
print(bytes(str(data), "utf-8").decode("unicode_escape"))

I know that a common suggestion is to just use the Requests library, and I have tried this, but I specifically want to know how to do this without using Requests.

The behavior I am looking for can be replicated with the following code that successfully logs in to the site using Requests:

import requests
payload = {"username": "USERNAME-HERE",
           "password": "PASSWORD-HERE",
           "redirect": "index.php",
           "sid": "",
           "login": "Login"}
p = requests.Session().post('https://osu.ppy.sh/forum/ucp.php?mode=login', payload)

# print the HTML after the request
print(p.text)

It seems to me that the http.client code is not "delivering" the payload, while the Requests code is.

Any insights? Am I overlooking something?

---

EDIT: Adding conn.set_debuglevel(1) gives the following output:

send: b'POST /forum/ucp.php?mode=login HTTP/1.1
Host: osu.ppy.sh
Accept-Encoding: identity
Content-Length: 70'


send: b'redirect=index.php&sid=&login=Login&username=USERNAME-HERE&password=PASSWORD-HERE'
reply: 'HTTP/1.1 200 OK'


header: Date
header: Content-Type
header: Transfer-Encoding
header: Connection
header: Set-Cookie
header: Cache-Control
header: Expires
header: Pragma
header: X-Frame-Options
header: X-Content-Type-Options
header: Server
header: CF-RAY

Answer 1

since you are urlencoding your payload, you must send the header: application/x-www-form-urlencoded