Search code examples
web-scraping

When should one go for webscraping tools? What is the advantage of having web scraping tool?

I am trying to learn about web scraping tools.

So if anyone help me on getting it started some tutorial links may help.

  1. When should one go for web scraping

  2. What are the benefits over rss feed?

  3. Best tools available in market for web scraping

Thanks!

Solution

  • To make the essentials short: "That depends on what you're trying to achieve."

    If you have an RSS-feed available with all the information that you need, you don't need to scrape a web page.

    If you're trying to extract data from a website that does not provide an API to access the data directly, you can use scraping to extract the information that you want from the page in a structured way. You can save the data into a database and work from there.

    For example: In the early Web 2.0 times, there were sites which scraped all other "flight"-pages to extract the cheapest flight for a given source and destination.