Search code examples
javaeclipseurlbots

How do I code an automated bot that can browse and do operations on a webpage?

I need to code a bot that needs to do the following:

Go to a JSP page and search for something by:

  1. writing something on a search box
  2. clicking the search button (submit button)
  3. clicking one of the the resulting buttons/links (same JSP page with different output)
  4. get the entire HTML of the new page (same JSP page with different output)

The 4th one can be done with screen scraping and I do not think I need help with it. But I need some guidance to do the options from 1 to 3. Any links or just some keyword that will help me Google to learn about it will be appreciated. I plan to do this with Java.

Solution

  • My suggestion is to use Selenium (http://docs.seleniumhq.org/download/). Install Selenium IDE in your firefox, and it can record what you do on a website, store it into a script and reply it. This video (http://www.youtube.com/watch?v=gsHyDIyA3dg) is gonna be helpful if you are a beginner.

    And if you want to do it in Java, its easy, just export the scripts in Selenium IDE to JUnit Webdriver code.

    Of course you can use Selenium Java webdriver in Java to write your program to operate on website directly.