Search code examples
pythonbeautifulsoupencodingrequest

python response.text cant decode

import requests
from bs4 import BeautifulSoup

url = "https://site"

paramsRequest = {

}

headersRequest = {
"Content-Type": "text/html; charset=utf-8",
}

response = requests.get(url, params=paramsRequest, headers=headersRequest)

file_path = 'D:/tempfiles/file.txt'

with open(file_path, 'w', encoding='utf-8') as file:
    file.write(response.text)

with open(file_path, 'r', encoding='utf-8') as file:
    file_content = file.read()

print(file_content)
soup = BeautifulSoup(response.text, 'html.parser') 
print(soup.prettify())

response.encoding = response.apparent_encoding
print(response.text)

try this code... but got answer like this:

�� ��9�?���/����@�[��08�/��)�#�h4�f���A��ѸQ(M{�B���������4�k(i76���r0� ;.�b>�s�$g�& 8��x�G7E��88&�s�jܦ��+�7��&)���1�������� )�~,�d���� }c j������?�$�e�T��K�K�4��c�v:�K���)�TJ �X����H5�é�ߏ�r)�:����_���

enter image description here

I tried

  1. response.encoding = response.apparent_encoding
  2. soup = BeautifulSoup(response.text, 'html.parser') print(soup.prettify())

upd: FULL Request=

Accept: application/json, text/plain, */*

Accept-Language:en-US;q=0.5,en;q=0.3

Accept-Encoding: gzip, deflate, br

Content-Type: text/html; charset=utf-8

X-Lead-Source: website

Connection: keep-alive

Cookie: ew_source=widget

Sec-Fetch-Dest: empty

Sec-Fetch-Mode: cors

Sec-Fetch-Site: same-origin

TE: trailers

answer:json

Solution

  • I was confused when I got binary file in python script, but same request in firefox mozilla works correct.

    So I removed header: "Accept-Encoding: gzip, deflate, br", and now it works correct. thx everyone