My scraper has been running fine from AWS ECS until recently, when I have been getting the following error message:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 114
Current browser version is 119.0.6045.199 with binary path /usr/bin/google-chrome
I know I am supposed to make the versions match, but I am not an expert in writing command line scripts, so I don't know how to fix the issue.
My current Dockerfile is
from python:3.10.11-bullseye
# install google chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get -y update --fix-missing
RUN apt-get install -y google-chrome-stable
# install chromedriver
RUN apt-get install -yqq unzip
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
# install requirements
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN pip install -r requirements.txt
ENTRYPOINT ["python", "app.py"]
The top portion of the app.py file is
# import relevant selenium packages
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
# set up headless option
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--allow-running-insecure-content')
chrome_options.add_argument('--ignore-certificate-errors')
# import support packages
from datetime import date
from datetime import datetime
import time
import re
import logging
import numpy as np
import pandas as pd
import traceback
# import AWS-related packages
import boto3
from io import StringIO
# initiate the Selenium driver
driver = webdriver.Chrome(chrome_options=chrome_options)
And the requirements.txt file is
# packages
selenium==4.8.2
boto3
numpy==1.23.2
pandas==1.4.3
Following @rasjani's advice in the comments section, I bumped the Selenium version to 4.14.0 and removed the chromedriver installation instructions from the Dockerfile. The scraper is working again.