Why is Selenium ChromeDriver throwing a "Connection aborted."/"RemoteDisconnected" error when running on a remote server but not on local?

Question

TL;DR:

Selenium is throwing this error

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

on the remote server but it runs perfectly on the local Docker instance.

---

I have Selenium/ChromeDriver running in Docker that works perfectly locally but I get a protocol error whenever I run the same container on a remote server (AWS Batch in this case).

My understanding of Docker is that it runs in its own environment so if it works locally it should work remotely. I've wiped any existing Chrome drivers on my laptop (including uninstalling Chrome) to double-check it wasn't using something available locally that isn't on the remote instance but that didn't change anything.

High-level stuff:

• Selenium version: v4
• ChromeDriver version (it always uses the latest version but as of now): 102.0.5005.61
• Python version: 3.9
• Remote machine - AWS Fargate instance with Public IP (more details can be provided if needed)

Note: Based on this SO answer, I know the issue is an incompatibility between the ChromeDriver and the Chrome browser but I can't see why that would change locally vs the remote!!

Code in question to reproduce:

Dockerfile

FROM python:3.9

# set a directory for the app
WORKDIR /app
RUN apt-get update 

RUN apt-get install -y wget xvfb unzip

# Set up the Chrome PPA
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list

# Set up Chromedriver Environment variables
ENV CHROMEDRIVER_DIR ./chromedriver
RUN mkdir $CHROMEDRIVER_DIR
RUN wget -O ./chromedriver/LATEST_RELEASE http://chromedriver.storage.googleapis.com/LATEST_RELEASE

# # Update the package list and install chrome
RUN apt-get update -y
RUN apt-get install -y google-chrome-stable=$(cat ./chromedriver/LATEST_RELEASE)-1

# # Download and install Chromedriver
RUN wget -q --continue -P $CHROMEDRIVER_DIR "http://chromedriver.storage.googleapis.com/$(cat ./chromedriver/LATEST_RELEASE)/chromedriver_linux64.zip"
RUN unzip $CHROMEDRIVER_DIR/chromedriver* -d $CHROMEDRIVER_DIR
# Put Chromedriver into the PATH
ENV PATH $CHROMEDRIVER_DIR:$PATH


RUN pip install --no-cache-dir pipenv

COPY Pipfile .

COPY Pipfile.lock .

RUN pipenv install --dev

COPY . .

CMD ["pipenv", "run", "main"]

Simplified python file:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def create_web_driver():
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("window-size=1400,2100")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--version")

    driver = webdriver.Chrome(
        # The driver location is inferred from the PATH variable
        options=chrome_options,
    )
    return driver

if __name__ == "__main__":
    driver = create_web_driver()
    # Error occurs on this line
    web_driver.get(url)

And the error I'm getting in AWS but not locally:

Traceback (most recent call last):
  File "/app/main.py", line 18, in <module>
    web_driver.get(url)
... (truncated for brevity)
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I've been trying to fix this bug for 4 days now, any advice or previous experience with this would be greatly appreciated!

Answer 1

I found the fix as soon as I posted the question, all I had to do was change this: chrome_options.add_argument("--disable-gpu")

to this:
chrome_options.add_argument("--disable-dev-shm-usage")

And it all worked perfectly!