Search code examples
javascripthtmlweb-scrapinghtmlunithtml-input

HTMLUnit HtmlTextInput and submitbutton


Trying to use HTMLUnit on a page called TimeEdit which is used to make schedule mainly for schools. I want to add a search term ("DV1431") in the inputfield here: TimeEdit

And then I want to submit this some way. I have read that you can trigger JavaScript with HTMLUnit and also that you can use buttons and even create "fake ones". But I'm not sure what is the best way in my case.

I have tried a solution that I found here on Stackoverflow: Youtube-example This one worked, but on my page TimeEdit there is no HTML-form that I can use. Instead there is just a class called searchButtons where the submit-button and input field is located. Here is my example-code:

    WebClient webClient = new WebClient(BrowserVersion.CHROME);
    webClient.getOptions().setCssEnabled(false);
    webClient.getOptions().setJavaScriptEnabled(true);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.getOptions().setUseInsecureSSL(true);
    webClient.getCookieManager().setCookiesEnabled(true);

    //Get the page
    HtmlPage currentPage = webClient.getPage("https://se.timeedit.net/web/bth/db1/sched1/ri1Q7.html");

    //Just for testing purpose
    System.out.println(currentPage.getUrl());

    // Get form where submit button is located
    // But on this page there is no form, just a class called searchButtons
   HtmlForm searchForm = (HtmlForm) currentPage.getElementById("ffsearchname");

    // Get the input field.
    HtmlTextInput searchInput = (HtmlTextInput) currentPage.getElementById("ffsearchname");

    // Insert the search term.
    searchInput.setText("DV1431");

    // Workaround: create a 'fake' button and add it to the form.
    HtmlButton submitButton = (HtmlButton) currentPage.createElement("button");
    submitButton.setAttribute("type", "submit");
    searchForm.appendChild(submitButton);

    // Workaround: use the reference to the button to submit the form. 
    HtmlPage newPage = submitButton.click();

    //Testing purpose
    System.out.println(newPage.getUrl());

How can I access the submit-button and input-field when it's not in a form and how do I submit when I can set my searchterm in the inputfield?


Solution

  • No idea why you try to insert a button and why you expect that button might be of any use. If you like to automate some Html pages you need some basic understanding of the way html and in your case also javascript works.

        // no need to set any options just use the default
        WebClient webClient = new WebClient(BrowserVersion.CHROME);
    
        // Get the page and wait for the javacode that will start with an minimal delay
        // doing something like setTimeout(init(), 10); is a common trick done by some js libs
        HtmlPage currentPage = webClient.getPage("https://se.timeedit.net/web/bth/db1/sched1/ri1Q7.html");
        currentPage.getEnclosingWindow().getJobManager().waitForJobsStartingBefore(100);
    
        // Get the input field.
        HtmlTextInput searchInput = (HtmlTextInput) currentPage.getElementById("ffsearchname");
    
        // Insert the search term.
        searchInput.setText("DV1431");
    
        // the output
        DomElement output = currentPage.getElementById("objectsearchresult");
        System.out.println("- before -------------------------------------------------------------------");
        System.out.println(output.asText());
        System.out.println("----------------------------------------------------------------------------");
    
        // try to find the button
        for (final DomElement elem : currentPage.getElementsByTagName("input")) {
            if ("Sök".equals(((HtmlInput) elem).getValueAttribute())) {
                // click and again wait for the javascript
                currentPage = elem.click();
                currentPage.getEnclosingWindow().getJobManager().waitForJobsStartingBefore(100);
    
                System.out.println();
                output = currentPage.getElementById("objectsearchresult");
                System.out.println("- after --------------------------------------------------------------------");
                System.out.println(output.asText());
                System.out.println("----------------------------------------------------------------------------");
                break;
            }
        }
    

    BTW: this code is only a sample to prove HtmlUnit will be able to automate your page. For finding controls you should study the various options the api offers and choose the appropriate one for your use case (maybe XPath).