Calling a script from a certain URL in Java?

Question

I'm using Java to parse HTML from a random website, let's say it's http://google.com for convenience. After parsing the HTML data, I want to extract some of that data, and show it on a display. After that the user will get to input a search term, and press a button. This button will execute that script behind the "search" button. I want to do this with several sites, so giving me a way that only works with google won't help me a lot.

Answer 1

Edit:

Ah, I see. You are asking about how to call a remote web-page from your code? There are a couple of ways you can do this:

• You can do it "by-hand" using the Java URL class.
• You could use the great Apache HTTPClient library.
• Another possibility is a tool like HTMLUnit.

---

Scraping of websites is a difficult problem and rarely have I found that a single scraper can handle multiple websites. The idea of a generic scraper is just not possible.

I would recommend writing a Java interface which is something like HandleSearchPage. It would contain a method to scrap the page and extract some of the data and another method to submit a search.

Then you can implement your scrapers for Google, Yahoo, etc.. As to how to parse html and drive a webpage there are many other questions/answers on that specific topic.

• Some help scraping a page in Java
• Trying to parse links in an HTML directory listing using Java

Best of luck.