Search code examples
javaauthenticationhtmlunit

Getting info from a page that requires Login (Java)

I'm making a small script that's supposed to get roughly 300 links from a page and make them into shortcuts (all saved in a folder).

I am able to get all the links I need from some pages, but some are part of a website that needs me to log in first.

I tried HttpUnit, but i simply fails every time. Until now I just put the Html page into an inputStream and read from there (line by line, looking for what I need), but I have no clue how to connect to the website or do anything else once I get to the Login part.

Here's the HttpUnit code, if it helps anyone:

final WebClient webClient = new WebClient();

// Get the first page
final HtmlPage page1 = webClient.getPage("mywebsite");

ArrayList<HtmlForm> f;
f = (ArrayList<HtmlForm>) page1.getForms();

System.out.println(f);

// Get the form that we are dealing with and within that form, 
// find the submit button and the field that we want to change.
final HtmlForm form = page1.getFirstByXPath("//form[@id='login']");

final HtmlSubmitInput button = form.getFirstByXPath("//input[@value='Login']");
final HtmlTextInput username = form.getFirstByXPath("//input[@id='username']");

// Change the value of the text field
username.setValueAttribute("username");

final HtmlPasswordInput passField = form.getFirstByXPath("//input[@id='password']");

// Change the value of the text field
passField.setValueAttribute("pass");

// Now submit the form by clicking the button and get back the second page.
final HtmlPage page2 = button.click();

webClient.closeAllWindows();

please forgive my bad variable naming :p It's a script just for myself, so I didn't really bother.

I get a NullPointerException on "final HtmlPage page2 = button.click();"

Thanks in advance.

Solution

  • It appears that your search for the button fails. After this line

    final HtmlSubmitInput button = form.getFirstByXPath("//submit[@value='Login']");

    I would add

    assert(button != null) : "Could not find the button";

    And run your app with assertions on (-ea parameter to the JVM) and it will report an assertion failure right there.