I am trying to retrieve app related information from Google Play store using selenium and BeautifulSoup. When I try to retrieve the information, I got webdriver exception error. I checked the chrome version and chrome driver version (both are compatible). Here is the weblink that is causing the issue, code to retrieve information, and error thrown by the code:
Code:
driver = webdriver.Chrome('path')
driver.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')
soup = bs.BeautifulSoup(driver.page_source, 'html.parser')
I am getting error on third line. Here is the parts of the error message:
Start of the error message:
---------------------------------------------------------------------------
WebDriverException Traceback (most recent call last)
<ipython-input-280-4e8a1ef443f2> in <module>()
----> 1 soup = bs.BeautifulSoup(driver.page_source, 'html.parser')
~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py in page_source(self)
676 driver.page_source
677 """
--> 678 return self.execute(Command.GET_PAGE_SOURCE)['value']
679
680 def close(self):
~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
318 response = self.command_executor.execute(driver_command, params)
319 if response:
--> 320 self.error_handler.check_response(response)
321 response['value'] = self._unwrap_value(
322 response.get('value', None))
~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
WebDriverException: Message: unknown error: bad inspector message:
End of the error message:
(Session info: chrome=79.0.3945.117)
Could anyone guide me how to fix the issue?
I think this is due to the chromedriver encoding problem. See https://bugs.chromium.org/p/chromium/issues/detail?id=723592#c9 for additional information about this bug.
Instead of selenium you can get page source using BeautifulSoup as follows.
import requests
from bs4 import BeautifulSoup
r = requests.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')
soup = BeautifulSoup(r.content, "lxml")
print(soup)