Unable to read wiki page by BeautifulSoup

Question

I tried to read wiki page using urllib and beautiful soup as follows.

I tried according to this.

import urllib.parse as parse, urllib.request as request
from bs4 import BeautifulSoup

name = "メインページ"
root = 'https://ja.wikipedia.org/wiki/'
url = root + parse.quote_plus(name)

response = request.urlopen(url)
html = response.read()
print (html)

soup = BeautifulSoup(html.decode('UTF-8'), features="lxml")
print (soup)

The code run without error but could not read Japanese characters.

Answer 1

Your approach seems correct and working for me. Try printing soup parsed data using following code and check the output.

soup = BeautifulSoup(html.decode('UTF-8'), features="lxml")
some_japanese = soup.find('div', {'id': 'mw-content-text'}).text.strip()
print(some_japanese)

In my case, I am getting the following(this is part of the output) -

ウィリアム・バトラー・イェイツ（1865年6月13日 - 1939年1月28日）は、アイルランドの詩人・劇作家。幼少のころから親しんだアイルランドの妖精譚などを題材とする抒情詩で注目されたのち、民族演劇運動を通じてアイルランド文芸復興の担い手となった。……

If this is not working for you, then try to save html content to file, and check the page in browser, if japanese text is fetching properly or not. (Again, its working fine for me)