Search code examples
pythonurllib

having encoding issue while using a url to request via urllib python

I am trying to use urllib to get a response from URLs and then using bs4 to extract data but there are some URLs on which I am getting this exception and 

from urllib.request import urlopen

is failed to process the request. Given below error:

Edited: Error:(traceback complete)

Traceback (most recent call last):
  File "G:/Internships/Botnostic Solutions/AllScrapers/careerz360/test_bs4.py", line 74, in <module>
    html = urlopen(url_link)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1240, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1107, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 18: ordinal not in range(128)

Code snippet:

from bs4 import BeautifulSoup
from urllib.request import urlopen
urls_list = []
with open('links_file.txt', 'r', encoding="utf-8") as linksFile:
    for l in linksFile:
        # url_parsed = urllib.parse.quote(l.strip())
        urls_list.append(l.strip())
linksFile.close()

Appending all links to a list so that can process them further using: urlib.urlopen()

This is the link which is creating the issue here:

urls_list = ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']

Remianing code:

for url in urls_list:
    html = urlopen(url)
    if html.getcode() == 200:
        response = BeautifulSoup(html.read(), "lxml")
        company_name = response.select('h2#business-name')[0].text.strip()

Different encodings used:

I have also tried for different encodings like: latin-1, utf-8 and iso etc, but getting the same error. Please suggest me a solution if possible. Thanks and regards

Solution

  • This code works for me. In file.txt I have one URL, https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853:

    import requests
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

soup = BeautifulSoup(requests.get(urls_list[0]).text, 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

    Prints:

    ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
Protègè Global

    I'm using Python 3.6.8

    EDIT: With urlopen you need to quote() the URL:

    from urllib.request import urlopen
from urllib.parse import quote, urlparse
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

p = urlparse(urls_list[0])
url = p.scheme + '://' + p.netloc + quote(p.path)

print(url)

html = urlopen(url)
soup = BeautifulSoup(html.read(), 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

    Prints:

    ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
https://www.careerz360.com/pakistan/prot%C3%A8g%C3%A8-global-android-developer-karachi-jobs-108853
Protègè Global