How to log into a website that uses a CSRF token using the python requests library

Question

website: https://auth.pleaseignore.com/login/?next=/profile/

import requests
from bs4 import BeautifulSoup

request_url = 'https://auth.pleaseignore.com/login/'
with requests.session() as session:
    get_url = session.get('https://auth.pleaseignore.com/login/')
    HTML = BeautifulSoup(get_url.text, 'html.parser')
    csrfmiddlewaretoken = HTML.find_all('input')[-1]['value']

    #logging in
    payload = {
    'next' : '/ profile /',
    'username' : 'asfasf',
    'password' : 'afsfafs',
    'next': '/ profile /',
    'csrfmiddlewaretoken': csrfmiddlewaretoken
    }
    login_request = session.post(request_url,payload)
    print(login_request)

Output:

<Response [403]>

The reason that I am getting a 403 response is because the csrfmiddlewaretoken token is invalid and the reason it's invalid is because the csrfmiddlewaretoken token changes every time a .get and .post request is sent, and I was wondering how I can log into the website despite that

Answer 1

The missing 'Referer' header are causing the [403 Forbidden].

headers = {'Referer': 'https://auth.pleaseignore.com/login/'}
login_request = session.post(request_url,payload, headers=headers)

Full script:

import requests
from bs4 import BeautifulSoup

request_url = 'https://auth.pleaseignore.com/login/'
with requests.session() as session:
    get_url = session.get('https://auth.pleaseignore.com/login/')
    HTML = BeautifulSoup(get_url.text, 'html.parser')
    csrfmiddlewaretoken = HTML.find_all('input')[-1]['value']

    #logging in
    payload = {
        'next' : '/ profile /',
        'username' : 'asfasf',
        'password' : 'afsfafs',
        'next': '/ profile /',
        'csrfmiddlewaretoken': csrfmiddlewaretoken
    }
    headers = {
        'Referer': 'https://auth.pleaseignore.com/login/'
    }
    login_request = session.post(request_url,payload, headers=headers)
    print(login_request)