Can google app engine be used running a web crawler?
I have a crawler process that starts with a specific id (or ASIN number) of an amazon web store product. The process then uses amazon product advertising API to get similar products. After the similar products are fetched, their ASIN numbers are used to call the API for other similar products. This process is suppose to get the information of about 25,000 products which roughly takes 3-6 hours.
Now my client wants to run multiple instances of this process with different initial ASIN numbers. I gave him a proposal on how the whole system can be made using google compute engines, but he is eager to use google app engine. He thinks that app engine will somehow automatically make crawler instances.
Since google app engine is used for backends and APIs, I have concerns that deploying such a crawler system on app engine would have its limitations.
Yes. You could fetch the base URL, scrape the data, get the urls for related products, then spin up taskqueues to fetch each subsequent url. Rinse and repeat.