Search code examples
web-scrapingweb-crawlerapify

Using Apify, how to extract data that is dependent on a modal window input?

I am interested in crawling a website with a modal window to select the location. The website data is dependent on the location selected on the modal window.

How can I display the modal window and select the location I am interested to crawl?

I am familiar with Apify and I am planning to use the interceptRequest() callback function to display the modal window and select it, but I am not very familiar on how to do it.

Edit: Adding picture for clarity enter image description here

enter image description here

Solution

  • Usually, modals are not so special in scraping/automation. You can wait for the selector of the modal to load and then you can click to close the modal or do any other action.

    So I checked the workflow and there is nothing special. I will not include the whole code. You can find detailed tutorials on Apify SDK and docs of Puppeteer where you can find all its functionality online.

    So once you are in your handlePageFunction in either a raw actor or Puppeteer Scraper, you can simply click on the Ver Precio button, wait for the modal to open and then fill the inputs and submit it.

    await page.click('.verprecio');
await page.waitForSelector('#frmBuscarSucursal2'); // Selector of the modal body
// Alternatively, if this will cause problems, you can just use plain waitFor
// await page.waitFor(5000) // waits 5 secs

// Now we select region
await page.select('#cmbRegiones2', '11'); // Each region has a number value

await page.select('#cmbMunicipio2', '901'); // The same for municipio

// And we submit by clicking on Buscar
await page.click('#btn-modal-buscar2');

    If you want to use Web Scraper for simplicity, you can also manage this with pure javascript but it is usually less reliable than using Puppeteer functions.