Search code examples
python-3.xbeautifulsoupurllib

Yet another Python encoding issue

I have trying this code:

def process_request(url):
    req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    return urlopen(req).read()

def get_links():
    url = c.first_url
    html = process_request(url)
    details_pages = []
    soup = BeautifulSoup(html, 'html.parser')
    links = soup.select(".pagelist-bar  a")
    print(links)
    for l in links:
        print(l)
        if l.has_attr('href'):
            href_ = l['href']
            detail = c.base_url + href_
            logging.info("Page with List of persons: %s", detail)
            details_pages.append(detail)
    return details_pages

def person_urls():
    pages = get_links()
    for l in pages:
        print("link: %s", l)
        doc = process_request(l)
        soup = BeautifulSoup(doc, 'html.parser')
        fichas = soup.select(".ficha")
        print(fichas)

In this url: http://www.guardiacivil.es/es/colaboracion/buscados/index.html

And no matter what strategy I use, this line: 

<a href="/es/colaboracion/buscados/index.html?buscar=si&category=abcd&notshown=">

is always converted to:

<a href="/es/colaboracion/buscados/index.html?buscar=si&category=abcd¬shown=">

The &notshown= becomes ¬shown=

I have tried some of the suggestions on THESE POSTS with no results so far.

Besides always having the error:

  self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1250, in _send_request
  self.putrequest(method, url, **skips)
  File "/usr/lib/python3.6/http/client.py", line 1117, in putrequest
  self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xac' in position 69: ordinal not in range(128)

Can anyone help me pls?

Solution

  • Maybe you should try replacing html.parser by just html in your BeautifulSoup call :

    soup = BeautifulSoup(html, 'html')
links = soup.select(".pagelist-bar  a")
#Ouptut
for x in links:
    print(x.get('href'))

    Output:

    /es/colaboracion/buscados/index.html?pagina=1&buscar=si&category=&notshown=
/es/colaboracion/buscados/index.html?pagina=2&buscar=si&category=&notshown=
/es/colaboracion/buscados/index.html?pagina=3&buscar=si&category=&notshown=
/es/colaboracion/buscados/index.html?pagina=4&buscar=si&category=&notshown=