How would you build a price comparison script? I know Amazon offers a public API, but I saw these two sites goodreads, bookdope which compare book prices, retrieve prices from Walmart and others websites that do not offer APIs. How do you get prices from sites that do not have an API?
I'm using C# and ASP.NET MVC.
If the site does not offer an API then you need to do what is called screen scraping. You get the HTML of the page and you have your program extract the information out of it.
This works, but beware of the problems.
Companies ofen don't like you doing this. If they wanted you to get the information they'd have provided an API. They may even detect a high number of hits from your specific location and block you.
Above all else, the page is designed for a human to read in a browser. They can change it at any time and your application may fail as a result.