Search code examples
pythonpython-3.xcurlpython-requestscloudflare

Curl works but python requests doesnt (curl gives 304 and 200, requests a 403). Datadome, Cloudflare?

When I go to vinted.nl a cookie is set. This allows me to do an api call in the browser and i can also do it in curl, but not using python requests. I am also wondering if this is because they use Datadome to prevent certain http requests? Or is it just a mistake in my python requests code? Or is it CloudFlare causing this?

I've tried VPN, home network, work network etc. Always this response when i use python requests, but when i use curl or postman i do get a correct response from the api.

This works:

curl 'https://www.vinted.nl/api/v2/languages' \
  -H 'cookie: anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c; v_udt=dFpFa0JZbDhLcnpXTzBMRGNHQ1Qwa3RGZm1vNFg0aVhOM29WTmc9PS0tVXIrVEtZRDNhMFdVMnhnbS0tVE1qVVVUeHRFMEk3LzVESlVwUEZ0UT09; v_sid=2e6ae0379a464a29cd97ead68aae344f; ab.optOut=This-cookie-will-expire-in-2024; _pbjs_userid_consent_data=7944749324711140; __cf_bm=dYupWIvIxLP6czwXium_JL.Sa1ALZDQydFWupYQS2lc-1689514518-0-Aev5HXwCkVwTwm5ecWnXi203ANDsvXiys1gB77le1YfP7/taQfDKbSTZadENhutigSewOGb98KaDPQrzXAIwJQYsYnIhckhdrhgaUdvJ3pkogtE6Jwxqiy2dZrbksApOrA==; viewport_size=374; _vinted_fr_session=b1psTUhzYXE0cjEwMGw4ck9CNWlVeEVtcG5kRTRES1dneHE0c3pEOEtxbUxNM0ZpTFdSS0NpZDdBdSs0a0FnR0RsZmtRL25wbDJBNjV0NFBKZzkwTWhBQzB4bndtNEhsWUFGVEpHM2xYRkY2K2ozcEhsUVNDdGgraHJBdlZVSVlRNUc0WHRHb0tidEFLQmI3NFovVEVTQUtDWnJvNkFKdlloVTRrL01UdjU3SW9vdWxmT3Y5RG12dTYvNGNMQmtxdkdyTHRMdUZuYWc0VW1vdGJYZGZydnFrbURUZFBHa2Z1b040VHM2WXpGT0dvbzJNZEFmY3FTYWVXcFR5OS9BYXZuMFU1NFpLTjY3WW83QlBlTjVNL0pDWm5WTXVoRmQ3Zm5Obmh3SjdsOTdrTU5rWjd5MGNydFlBVVJ1RmJjKzhFZkNEMy81MEoyeDB2Zm5kdGRMcmMzaElYSlkyNW9nTjZWRks2VXpjUVp2WW9TY3dDOWFtVnNNVDBabHhzZXhvZS9HQ2p4NTh6VHdKTWhCajVpSlRZQUlZdXBlZXVEUFlCaSt1ejU4U1pLUTNMMVNiTmR4ZEdBa1RKMDBLTUVJV0JrYkdmaStoN3BEcWNSWWgwdCttNE1Gd0FOSG9aeS9mRTFRZmhSaStZN3A5TXhiOXJjdE5LWWxPNzBBK29ydHl5MDhDbEd6RlU5OVJLYS95cVBNLzhnRWRjRlV0U00vL3RzYVZjdGVlY1dWTVl1cURLa1BjWnFabm5Ga01vUUNOYzMxUTRua3lyKyt3d2c1SlR5SU1KS3UvT3N3YUxsYzUzWURXQy95Q2V6TUpXeE1hdXk5eUR5dGFUSWZpYStLeEZLcU13NHBFcW9CT2d0VEZMWGZQbjdDMnFMV21FMk5WRmRxclhGYWlYaFNxa0lQeW9OYXpQSmpjVGQ5bGx5WmhaUGN0NjMxVmRNUDViTWE2ZnJ1bUlhNkh0R09FeCtJSCtrMERrTnVoRGFJWkhPc1JLa2x4QUlyb1dwYk1wSWJmNEM1ZGFSQlNVaGs2QndYRnBZZC90Z29ZNTZ2UXZVWWVHWFg1TVA3ZDY2RDFhbzJjKzVKMU04d0JyWEZQWUtsdlpFanAwRSt1R0FiR0EwTklCeDJyM0Mwc25LUktTMzRXNU1VaU45RGJweWdPNHJyUGx3UkxUenJ5Sy9PcndZb1JBemhqWUJUZTl4bGJadGtWUjEzSHRYNjRodkNETU9MS1ZTWXBkUTdQQ21OdDdTdWMvNzZLazRQY0ZiZlJpRXFjVHFTQWZJZ3B0RXdCS2dTTWRvb0J4RUFhTzZ2cFBKM0JpNFpNZnpValFYZ29va0FtRGgvZVc0VWhCNkd5aGFSVXJNVUZmRDEwVmNxbkMwU05QeW8vSXVpV3AvOER3cEd4Kzk5K0tISndNWkZJTzFpRGVGQXJVZmFxd1lpb1c0ZFllakUzYVNVQmswUDFzVFNRNmNDVkFYRUVRWjlPTnY2QXN0Zmw0MC94S3Ewd0JOR2gyNCtONjhQNnVtOXNodi9TbThLSUhhbEduM1h5N0IxZjg0UUh0NDFrMHhvejJkZUtKVHBUQ3hjdk1DST0tLVJnaVJLemdFamhpZ1pZQWphd3dpaUE9PQ%3D%3D--3c0df38decfa6be41b27f65b0f8073cf25f84a48; datadome=2~IXr6AtVrQ0uioqk1ZON1BWpEibj33CUCIOS8wuwAYvBWYjeE60K14p5Njb3e4VqYk2k3NOFnLFTDgRwSX4fC9dCfPRXpwgksoLXck2Y56oUZ2yK~nbLHpZdGyOnZh4; OptanonConsent=isGpcEnabled=0&datestamp=Sun+Jul+16+2023+15%3A49%3A51+GMT%2B0200+(Central+European+Summer+Time)&version=202305.1.0&browserGpcFlag=0&isIABGlobal=false&consentId=2ec2c37b-7d7a-4e65-b010-f9779852300c&hosts=&interactionCount=0&landingPath=https%3A%2F%2Fwww.vinted.nl%2F&groups=C0001%3A1%2CC0002%3A0%2CC0003%3A0%2CC0004%3A0%2CSTACK42%3A0%2CC0015%3A0; _dd_s=rum=0&expire=1689516288792'

I saw that curl sets some additional default headers, so i added those also to my python request: 
> GET /api/v2/languages HTTP/2
> Host: www.vinted.nl
> user-agent: curl/7.88.1
> accept: */*
> cookie: anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c etc etc

However when i try this with python requests, I get a 403:

import requests

cookie = "anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c; v_udt=dFpFa0JZbDhLcnpXTzBMRGNHQ1Qwa3RGZm1vNFg0aVhOM29WTmc9PS0tVXIrVEtZRDNhMFdVMnhnbS0tVE1qVVVUeHRFMEk3LzVESlVwUEZ0UT09; v_sid=2e6ae0379a464a29cd97ead68aae344f; ab.optOut=This-cookie-will-expire-in-2024; _pbjs_userid_consent_data=7944749324711140; __cf_bm=dYupWIvIxLP6czwXium_JL.Sa1ALZDQydFWupYQS2lc-1689514518-0-Aev5HXwCkVwTwm5ecWnXi203ANDsvXiys1gB77le1YfP7/taQfDKbSTZadENhutigSewOGb98KaDPQrzXAIwJQYsYnIhckhdrhgaUdvJ3pkogtE6Jwxqiy2dZrbksApOrA==; viewport_size=374; _vinted_fr_session=b1psTUhzYXE0cjEwMGw4ck9CNWlVeEVtcG5kRTRES1dneHE0c3pEOEtxbUxNM0ZpTFdSS0NpZDdBdSs0a0FnR0RsZmtRL25wbDJBNjV0NFBKZzkwTWhBQzB4bndtNEhsWUFGVEpHM2xYRkY2K2ozcEhsUVNDdGgraHJBdlZVSVlRNUc0WHRHb0tidEFLQmI3NFovVEVTQUtDWnJvNkFKdlloVTRrL01UdjU3SW9vdWxmT3Y5RG12dTYvNGNMQmtxdkdyTHRMdUZuYWc0VW1vdGJYZGZydnFrbURUZFBHa2Z1b040VHM2WXpGT0dvbzJNZEFmY3FTYWVXcFR5OS9BYXZuMFU1NFpLTjY3WW83QlBlTjVNL0pDWm5WTXVoRmQ3Zm5Obmh3SjdsOTdrTU5rWjd5MGNydFlBVVJ1RmJjKzhFZkNEMy81MEoyeDB2Zm5kdGRMcmMzaElYSlkyNW9nTjZWRks2VXpjUVp2WW9TY3dDOWFtVnNNVDBabHhzZXhvZS9HQ2p4NTh6VHdKTWhCajVpSlRZQUlZdXBlZXVEUFlCaSt1ejU4U1pLUTNMMVNiTmR4ZEdBa1RKMDBLTUVJV0JrYkdmaStoN3BEcWNSWWgwdCttNE1Gd0FOSG9aeS9mRTFRZmhSaStZN3A5TXhiOXJjdE5LWWxPNzBBK29ydHl5MDhDbEd6RlU5OVJLYS95cVBNLzhnRWRjRlV0U00vL3RzYVZjdGVlY1dWTVl1cURLa1BjWnFabm5Ga01vUUNOYzMxUTRua3lyKyt3d2c1SlR5SU1KS3UvT3N3YUxsYzUzWURXQy95Q2V6TUpXeE1hdXk5eUR5dGFUSWZpYStLeEZLcU13NHBFcW9CT2d0VEZMWGZQbjdDMnFMV21FMk5WRmRxclhGYWlYaFNxa0lQeW9OYXpQSmpjVGQ5bGx5WmhaUGN0NjMxVmRNUDViTWE2ZnJ1bUlhNkh0R09FeCtJSCtrMERrTnVoRGFJWkhPc1JLa2x4QUlyb1dwYk1wSWJmNEM1ZGFSQlNVaGs2QndYRnBZZC90Z29ZNTZ2UXZVWWVHWFg1TVA3ZDY2RDFhbzJjKzVKMU04d0JyWEZQWUtsdlpFanAwRSt1R0FiR0EwTklCeDJyM0Mwc25LUktTMzRXNU1VaU45RGJweWdPNHJyUGx3UkxUenJ5Sy9PcndZb1JBemhqWUJUZTl4bGJadGtWUjEzSHRYNjRodkNETU9MS1ZTWXBkUTdQQ21OdDdTdWMvNzZLazRQY0ZiZlJpRXFjVHFTQWZJZ3B0RXdCS2dTTWRvb0J4RUFhTzZ2cFBKM0JpNFpNZnpValFYZ29va0FtRGgvZVc0VWhCNkd5aGFSVXJNVUZmRDEwVmNxbkMwU05QeW8vSXVpV3AvOER3cEd4Kzk5K0tISndNWkZJTzFpRGVGQXJVZmFxd1lpb1c0ZFllakUzYVNVQmswUDFzVFNRNmNDVkFYRUVRWjlPTnY2QXN0Zmw0MC94S3Ewd0JOR2gyNCtONjhQNnVtOXNodi9TbThLSUhhbEduM1h5N0IxZjg0UUh0NDFrMHhvejJkZUtKVHBUQ3hjdk1DST0tLVJnaVJLemdFamhpZ1pZQWphd3dpaUE9PQ%3D%3D--3c0df38decfa6be41b27f65b0f8073cf25f84a48; datadome=2~IXr6AtVrQ0uioqk1ZON1BWpEibj33CUCIOS8wuwAYvBWYjeE60K14p5Njb3e4VqYk2k3NOFnLFTDgRwSX4fC9dCfPRXpwgksoLXck2Y56oUZ2yK~nbLHpZdGyOnZh4; OptanonConsent=isGpcEnabled=0&datestamp=Sun+Jul+16+2023+15%3A49%3A51+GMT%2B0200+(Central+European+Summer+Time)&version=202305.1.0&browserGpcFlag=0&isIABGlobal=false&consentId=2ec2c37b-7d7a-4e65-b010-f9779852300c&hosts=&interactionCount=0&landingPath=https%3A%2F%2Fwww.vinted.nl%2F&groups=C0001%3A1%2CC0002%3A0%2CC0003%3A0%2CC0004%3A0%2CSTACK42%3A0%2CC0015%3A0; _dd_s=rum=0&expire=1689516288792"

headers = {
    "Host": "www.vinted.nl",
    "User-Agent": "curl/7.88.1",
    "Accept-Encoding": None,
    "Connection": None,
    "accept": "*/*",
    "cookie": cookie,
}
url = "https://www.vinted.nl/api/v2/languages"

response = requests.get(url=url, headers=headers)

print(response)

The cookie probably expires, so you probably have to get the cookie value from your browser developer tools if you want to try my code.

This is a screenshot of how i get the cookie using Chrome dev tools: chrome dev tools Copy as Curl vinted.nl

response.status_code: 403

response.text:

  <h1>Access denied</h1>
  <p>You do not have access to www.vinted.nl.</p><p>The site owner may have set restrictions that prevent you from accessing the site.</p>
  <ul class="cferror_details">
    <li>Ray ID: 7e90b3ed3aa9b8c6</li>
    <li>Timestamp: 2023-07-19 05:53:13 UTC</li>
    <li>Your IP address: 86.etc</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: www.vinted.nl/api/v2/languages </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_521F90</li>
    <li>User-Agent: Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36</li>
  </ul>

response.headers

{'Date': 'Wed, 19 Jul 2023 05:53:13 GMT', 
'Content-Type': 'text/html; charset=UTF-8', 
'Transfer-Encoding': 'chunked', 
'Connection': 'keep-alive', 
'X-Frame-Options': 'SAMEORIGIN', 
'Referrer-Policy': 'same-origin', 
'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 
'Set-Cookie': '__cf_bm=LPhOJ80GUwCYRjFvS6m.8i.7NVl_YXcuEJufH5WHhrQ-1689745993-0-AWu4bkuqNL4hf1bx/IwLfM/RZY9dzgtyBMXq6m/gLv+s+nUHb1Lbc5Cw3/XPcVX9OnUAQURGbclo/OI7TyglbdEdVQr7S5r0Yq334L15kkKx; path=/; expires=Wed, 19-Jul-23 06:23:13 GMT; domain=.vinted.nl; HttpOnly; Secure; SameSite=None',
 'Vary': 'Accept-Encoding', 
'Server': 'cloudflare', 
'CF-RAY': '7e90b3ed3aa9b8c6-AMS', 
'Content-Encoding': 'gzip'}

response.request.headers:

{
'User-Agent': 'Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36', 
'Accept-Encoding': 'gzip, deflate', 
'Accept': '*/*', 
'Connection': 'keep-alive', 
'host': 'www.vinted.nl', 
'Cache-Control': 'no-cache', 
'cookie': '_vinted_fr_session=MElmUmdUcmNOb etc etc'}
Solution

  • So it was:

    • not the vpn (ip),
    • not behind a firewall,
    • not work network vs home network,
    • not the specific headers etc,
    • not the cookie,
    • not the automatic headers that requests or curl sets,
    • not deleting all existing vinted.nl cookies.

    Solved this by creating a virtual environment (using poetry) with a new python 3.11.4 and requests 2.31.0. After that it did all of a sudden work. Before this I was using the same python version and requests version, but the base installment. There must be something wrong there.

    I was also starting to think in this direction, but who knows what the actual problem is (python >=3.6 on OSX has no certificates at all, and can't validate any SSL connection): https://stackoverflow.com/a/42334357/3489155

    Could have also tried still to use urllib instead of requests as a fix.