how to manage multiple 'executable and datadir profile' for parallelizing the launch of scrappers?

Question

I am using , and have difficulties to launch 4 scripts at the same time. I have used theses variable for local browser

let CHROMIUM_DATA_DIR = `/Users/yo/dataDir/datadir${this.cmd}`
let CHROMIUM_EXEC_PATH = `/Applications/Google-Chrome${this.cmd}.app/Contents/MacOS/Google Chrome`

I have multiplied by 4, the same datadir, et the same executable. I have just renamed the files/directories.

It does not work well. What would be your recomendation, to quickly scale the launch of the scrappers (). How could I install various chromes instance, et managing according datadir (to save some login session etc..)

tks

Answer 1

Since you are using playwright, you can use persistent contexts.

You do not need to create your own data directories or executables by copying them, simply pass location of an empty directory when launching the browser and playwright will populate it itself, storing any session data.

I do not use node.js, but just to give an idea, sample code in python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch_persistent_context(user_data_dir=r'C:\Users\me\Desktop\dir', headless=False)

    page = browser.new_page()
    page.goto("http://playwright.dev")
    print(page.title())
    browser.close()