Search code examples
pythonbeautifulsoup

Turn off BeautifulSoup messy encoding confidence output

I'm using bs4 for my project. It print out messy output with many encoding confidence score whenever I create a soup instance:

req = urllib2.Request(url, headers=hdr)
page = urllib2.urlopen(req, timeout=5)
soup = BeautifulSoup(page.read(), "lxml")

It works fine, but with redundant output. I just want to remove it, but I can't find any information about something like verbose option. 

2018-11-15 10:40:46,286 utf-8  confidence = 0.99
2018-11-15 10:40:46,286 SHIFT_JIS Japanese confidence = 0.01
2018-11-15 10:40:46,287 EUC-JP Japanese confidence = 0.01
2018-11-15 10:40:46,287 GB2312 Chinese confidence = 0.01
2018-11-15 10:40:46,287 EUC-KR Korean confidence = 0.01
2018-11-15 10:40:46,287 CP949 Korean confidence = 0.01
2018-11-15 10:40:46,287 Big5 Chinese confidence = 0.01
2018-11-15 10:40:46,288 EUC-TW Taiwan confidence = 0.01
2018-11-15 10:40:46,288 windows-1251 Russian confidence = 0.01
2018-11-15 10:40:46,288 KOI8-R Russian confidence = 0.01
2018-11-15 10:40:46,288 ISO-8859-5 Russian confidence = 0.0
2018-11-15 10:40:46,288 MacCyrillic Russian confidence = 0.0
2018-11-15 10:40:46,288 IBM866 Russian confidence = 0.0
2018-11-15 10:40:46,289 IBM855 Russian confidence = 0.01
2018-11-15 10:40:46,289 ISO-8859-7 Greek confidence = 0.0
2018-11-15 10:40:46,289 windows-1253 Greek confidence = 0.0
2018-11-15 10:40:46,289 ISO-8859-5 Bulgairan confidence = 0.0
2018-11-15 10:40:46,289 windows-1251 Bulgarian confidence = 0.01
2018-11-15 10:40:46,290 TIS-620 Thai confidence = 0.0
2018-11-15 10:40:46,290 ISO-8859-9 Turkish confidence = 0.54363730033
2018-11-15 10:40:46,290 windows-1255 Hebrew confidence = 0.0
2018-11-15 10:40:46,290 windows-1255 Hebrew confidence = 0.0
2018-11-15 10:40:46,290 windows-1255 Hebrew confidence = 0.0
2018-11-15 10:40:46,291 utf-8  confidence = 0.99
2018-11-15 10:40:46,291 SHIFT_JIS Japanese confidence = 0.01
2018-11-15 10:40:46,291 EUC-JP Japanese confidence = 0.01
2018-11-15 10:40:46,291 GB2312 Chinese confidence = 0.01
2018-11-15 10:40:46,291 EUC-KR Korean confidence = 0.01
2018-11-15 10:40:46,291 CP949 Korean confidence = 0.01
2018-11-15 10:40:46,292 Big5 Chinese confidence = 0.01
2018-11-15 10:40:46,292 EUC-TW Taiwan confidence = 0.01

Please help. Any suggestion is gratefully appreciated!

Solution

  • You can set the log level higher like this:

    import logging
logger = logging.getLogger('chardet')
logger.setLevel(logging.CRITICAL)

    In general, if you want to find who produces some annoying logs, do the following:

    Provoke the logs to be emitted by running the code. In this case

    req = urllib2.Request(url, headers=hdr)
page = urllib2.urlopen(req, timeout=5)
soup = BeautifulSoup(page.read(), "lxml")

    Then the logger must be in this list

    import logging    
print(logging.Logger.manager.loggerDict.values())
[..., 'chardet', ...]

    Try switching off the loggers one by one. Once you do not see the logs any more, you know which is the log that emits it:

    import logging
for name in logging.Logger.manager.loggerDict.values():
    print(name)
    logger = logging.getLogger(name)
    logger.setLevel(logging.CRITICAL)
    # I have left the exact code here for demonstration purposes
    req = urllib2.Request(url, headers=hdr)
    page = urllib2.urlopen(req, timeout=5)
    soup = BeautifulSoup(page.read(), "lxml")

    Then set the log level before the code that emits the logs is run:

    import logging
logger = logging.getLogger('chardet')
logger.setLevel(logging.CRITICAL)
# No log output any more from here on
req = urllib2.Request(url, headers=hdr)
page = urllib2.urlopen(req, timeout=5)
soup = BeautifulSoup(page.read(), "lxml")