Search code examples
phpweb-scrapingweb-crawlerdata-extraction

Scrape and Extract data from https://nforlanwebdmz.phs.org when it is not visible in 'Source Code' of the webpage


I am trying to write an automated PHP script to scrape and extract all 'Job IDs' (3262, 3197, 3196 etc.) from URL https://nforlanwebdmz.phs.org/ltmprd/CandidateSelfService/controller.servlet?dataarea=ltmprd&context.session.key.HROrganization=90&context.session.key.JobBoard=EXTPHYS&context.session.key.noheader=true.

However, this does not seem to be straightforward because the required data is not directly visible in the source code of the webpage. I also tried inspecting 'Developer Tools->Network' of different browsers, however could not locate the source of the data.

Any help would be highly appreciated.

Thanks & Regards!

Developer Tools->Network


Solution

  • I take look at the Developer Tools>Network on Chrome, and found this API url: https://nforlanwebdmz.phs.org/ltmprd/soapExt/ldrest/JobPosting/JobPostingListWebServices_ListOperation?JobBoard=EXTPHYS&LocationOfJob=+&Category=+&WorkType=+&JobRequisition=+&Description_translation_=+&JobPosting=+&PostingStatus=2&PostingDateRange.Begin=+&PostingDateRange.End=+&JobRequisitionPriority=+&csk.IsoLocale=en&HROrganization=90&limit=-1&=1486230138234

    The "Job ID" is the "JobRequisition" in the JSON data