I want to use Playwright.connect() method using Proxy to consume Browserless. According to Browserless doc.
https://docs.browserless.io/docs/playwright.html
The standard connect method uses playwright's built-in browser-server to handle the connection. This, generally, is a faster and more fully-featured method since it supports most of the playwright parameters (such as using a proxy and more). However, since this requires the usage of playwright in our implementation, things like ad-blocking and stealth aren't supported. In order to utilize those, you'll need to see our integration with connectOverCDP.
I thought well connect will have a .setProxy(), Like launch()
browserType.launch(new BrowserType.LaunchOptions().setProxy(proxy));
But connect methods it has 2 variations
default Browser connect(String wsEndpoint) {
return connect(wsEndpoint, null);
}
Browser connect(String wsEndpoint, ConnectOptions options);
I thought well i will pick connect + ConnectOptions it sures has a .setProxy as well but it doesn't.
class ConnectOptions {
public Map<String, String> headers;
public Double slowMo;
public Double timeout;
public ConnectOptions setHeaders(Map<String, String> headers) {
this.headers = headers;
return this;
}
public ConnectOptions setSlowMo(double slowMo) {
this.slowMo = slowMo;
return this;
}
public ConnectOptions setTimeout(double timeout) {
this.timeout = timeout;
return this;
}
}
I have try this
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions().setProxy(proxy);
Browser browser = playwright.chromium()
.connect("wss://&--proxy-server=http://myproxyserver:1111")
.newContext(browserContextOptions)
.browser();
browser.newPage("resource");
But the proxy returns authentication is required.
I'm confused now Browserless says that .connect could provide a Proxy but how? Is browserless wrong? Or am I missing something? I'm new on this technology.
I have tried as well using page.setExtraHTTPHeaders.
private void applyProxyToPage(final Page page,final String
userPassCombination){
final String value = "Basic "+Base64.getEncoder().encodeToString(userPassCombination.getBytes(Charset.forName("UTF-8")));
page.setExtraHTTPHeaders(Collections.singletonMap("Authorization",value));
//page.setExtraHTTPHeaders(Collections.singletonMap("Proxy-Authorization",value));// Not working either
}
With the help of my friend Alejandro Loyola at Browserless, I am now able to connect. I will post the snippet:
private String navigateWithPlaywrightInBrowserlessWithProxy(final String token,final String proxyHost,final String userName,final String userPass,final String url){
final Browser.NewContextOptions browserContextOptions = new Browser.NewContextOptions()
.setProxy(new Proxy(proxyHost)
.setUsername(userName)
.setPassword(userPass));//Raw password not encoded in any way;
try (final Playwright playwright = Playwright.create(); Browser browser = playwright.chromium().connectOverCDP("wss://chrome.browserless.io?token=" + token);final BrowserContext context = browser.newContext(browserContextOptions);){
Page page = context.newPage();
page.route("**/*.svg", Route::abort);
page.route("**/*.png", Route::abort);
page.route("**/*.jpg", Route::abort);
page.route("**/*.jpeg", Route::abort);
page.route("**/*.css", Route::abort);
page.route("**/*.scss", Route::abort);
page.navigate(url, new Page.NavigateOptions()
.setWaitUntil(WaitUntilState.DOMCONTENTLOADED));
return page.innerHTML("body");
}
}
My gotchas were as follows.
I was using:
"wss://chrome.browserless.io/playwright?token=
Instead of:
"wss://chrome.browserless.io?token="
And use:
connectOverCDP