Opening web pages using search engine

Question

i am trying to open multiple web pages on the same domain. I need those links to scrape some data ( I have permission from the website). I cannot use the domain name that I actually want to scrape, but as an example: If a website is www.gsmarena.com, and all I want to scrape is all the pages on that website which ends with ends with a particular string: Example www.gsmarena.com/*anystring*/*searchstring*

I hope someone can help me out

Answer 1

I can tell you what I would do if I was in your situation, but you will need some experience creating scraping applications (my preferred language for scraping is C#.). Here are the steps(needs to be done using scraping application):

• Download main page as a string
• Get links to all categories(or the categories you need)(if you want to use C# then HTMLAgilityPack will help you to do this)
• Download Category by category and get their product link from there.

I hope it helped you. If you want to know more feel free to ask in comments.