Search code examples
windowsgoogle-chromeselenium-webdriverproxyfiddler

Accessing a website behind a proxy server with Google Chrome headless, using Selenium, which works when using Fiddler

I'm having trouble running a selenium test run, using google chrome headless, accessing a site which is only available through a proxy server.

TLDR: Chrome headless can't access sites through a proxy server unless fiddler is running

Trying to diagnose the problem, I've found a strange set of circumstances that cause it to work:

  • Internet explorer is configured to use the proxy and the site can be accessed only when the proxy is turned on. Without the Internet Explorer proxy set, there is no connectivity to the site whatsoever.
  • Chrome uses the Internet explorer proxy settings, so access to the site using chrome depends on the Internet explorer proxy being set.

  • As long as the Internet Explorer proxy is set:

    • When the Selenium tests are run with chrome (not headless), the site is accessed successfully.
    • When the Selenium tests are run with chrome (headless), the site is not accessed successfully (it times out).
    • If the proxy server configuration is passed through to chrome* (headless), the site is not accessed successfully.
    • If the proxy server configuration is not passed through to chrome (headless), the site is not accessed successfully.

  • If we install fiddler on the server, the proxy settings are replaced with the Fiddler proxy, and suddenly selenium tests running through chrome (headless) work successfully. And:

    • If the proxy server configuration is passed through to chrome* (headless), the site is accessed successfully.

    • If the proxy server configuration is not passed through to chrome (headless), the site is accessed successfully.

    • Proxy server configuration passed through to chrome using chrome_options.add_argument.

Notes: - There are no chrome extensions enabled.

  • The website is using HTTP, so there is no issue caused by certificates

  • Accessing the website via a route that doesn't need a proxy server, such as from our desktops, chrome headless works.

  • I've tried this on two different servers, with identical behaviour.

  • The proxy server doesn't require authentication.

So chrome headless can't access the sites through the proxy unless fiddler is running. Does anyone have an idea why this could be, and how I get it working without having Fiddler running?

Solution

  • I used network monitor to see that the headless chrome calls were resolving to a website IP address, rather than that of the proxy, so it looked like headless chrome wasn't using the default proxy, as suspected.

    In the end, configuring the nunit tests to pass the proxy server in to headless chrome using the --proxy-server argument did the trick.