Search code examples
javascriptseleniumweb-crawlerphantomjsscreen-scraping

Selenium with PhantomJS: Form being validated but not submitted


I'm having a strange problem submitting a form through Selenium Webdriver's PhantomJS API. Upon clicking the submit button, the form gets validated (are the username and password too short, or blank, etc.), but it does not get ultimately submitted. That is, if I submit an invalid form, and check the screenshot, there are alert notifications. If I submit a valid form, nothing happens. The JS on the page is supposed to validate the form, then submit it, when the submit button is clicked.

A couple rule-outs. I've tried both .click() and .submit() on the submit button, which is a button element. I've also tried .submit() on the form itself and on an arbitrary element (like password) within the form, which Selenium allows.

In fact, the same code works just fine if I'm driving Firefox and not PhantomJS. I'd rather not switch over, though, as Firefox is slower and gives me unpredictable connection trouble.

My specs: I'm using Selenium version 1.43 with PhantomJS 1.98 via Python 2.7 on Ubuntu 14.04 LTS (GNU/Linux 3.17.1-elastic x86_64).

Code below. First is my Selenium code. Then the html for the form. Then the login.js source code from the site. What I think is happening in the js is that for some reason the invalidHandler from the .validate function is being run, but the submitHandler is not? Thanks in advance for your attention.

# enter username and password up here first
submitEl = self.find_element_by_css_selector("button[type='submit']")
submitEl.click()
self.save_screenshot('login_submission.png')

Edit2: "self" is a driver object that inherits from the Webdriver.PhantomJS class. Now the form html:

            <form class="form-login" action="" name="login" method="POST">
                <div class="errorHandler alert alert-danger no-display">
                    <i class="fa fa-remove-sign"></i> You have some form errors. Please check below.
                </div>
                                    <fieldset>
                    <div class="form-group">
                        <span class="input-icon">
                            <input type="text" class="form-control" name="username" placeholder="Username">
                            <i class="fa fa-user"></i> </span>
                    </div>
                    <div class="form-group form-actions">
                        <span class="input-icon">
                            <input type="password" class="form-control password" name="password" placeholder="Password">
                            <i class="fa fa-lock"></i>
                            <a class="forgot" href="forgot.php">
                                I forgot my password
                            </a> </span>
                    </div>
                    <div class="form-group">
                    <img src="captcha/captcha.php" alt="captcha" />
                        <span class="input-icon" style="width:200px; float: right;">
                            <input type="text" class="form-control" name="captcha">
                            <i class="fa fa-key"></i> </span>
                    </div>
                    <div class="form-actions" ><div class="slideExpandUp">
                        <label for="remember" class="checkbox-inline">
                            <input type="checkbox" class="grey remember" id="remember" name="remember">
                            Keep me signed in
                        </label>
                        <button type="submit" class="btn btn-bricky pull-right" name="submit">
                            Login <i class="fa fa-arrow-circle-right"></i>
                        </button></div>
                    </div>
                    <div class="new-account">
                        Don't have an account yet?
                        <a href="register.php" class="register">
                            Create an account
                        </a>
                    </div>
                </fieldset>
            </form>

login.js

var Login = function () {
    var runSetDefaultValidation = function () {
        $.validator.setDefaults({
            errorElement: "span", // contain the error msg in a small tag
            errorClass: 'help-block',
            errorPlacement: function (error, element) { // render error placement for each input type
                if (element.attr("type") == "radio" || element.attr("type") == "checkbox") { // for chosen elements, need to insert the error after the chosen container
                    error.insertAfter($(element).closest('.form-group').children('div').children().last());
                } else if (element.attr("name") == "card_expiry_mm" || element.attr("name") == "card_expiry_yyyy") {
                    error.appendTo($(element).closest('.form-group').children('div'));
                } else {
                    error.insertAfter(element);
                    // for other inputs, just perform default behavior
                }
            },
            ignore: ':hidden',
            highlight: function (element) {
                $(element).closest('.help-block').removeClass('valid');
                // display OK icon
                $(element).closest('.form-group').removeClass('has-success').addClass('has-error').find('.symbol').removeClass('ok').addClass('required');
                // add the Bootstrap error class to the control group
            },
            unhighlight: function (element) { // revert the change done by hightlight
                $(element).closest('.form-group').removeClass('has-error');
                // set error class to the control group
            },
            success: function (label, element) {
                label.addClass('help-block valid');
                // mark the current input as valid and display OK icon
                $(element).closest('.form-group').removeClass('has-error');
            },
            highlight: function (element) {
                $(element).closest('.help-block').removeClass('valid');
                // display OK icon
                $(element).closest('.form-group').addClass('has-error');
                // add the Bootstrap error class to the control group
            },
            unhighlight: function (element) { // revert the change done by hightlight
                $(element).closest('.form-group').removeClass('has-error');
                // set error class to the control group
            }
        });
    };
    var runLoginValidator = function () {
        var form = $('.form-login');
        var errorHandler = $('.errorHandler', form);
        form.validate({
            rules: {
                username: {
                    minlength: 2,
                    required: true
                },
                password: {
                    minlength: 6,
                    required: true
                }
            },
            submitHandler: function (form) {
                errorHandler.hide();
                form.submit();
            },
            invalidHandler: function (event, validator) { //display error alert on form submit
                errorHandler.show();
            }
        });
    };
    return {
        //main function to initiate template pages
        init: function () {
            runSetDefaultValidation();
            runLoginValidator();
        }
    };
}();

Edit: fixed the title

PhantomJS log file output, by request:

PhantomJS is launching GhostDriver...
[INFO  - 2015-01-27T16:58:04.367Z] GhostDriver - Main - running on port 48152
[INFO  - 2015-01-27T16:58:05.366Z] Session [a9641420-a645-11e4-95fd-c78c9ec356b6] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0","webSecurityEnabled":true}
[INFO  - 2015-01-27T16:58:05.366Z] Session [a9641420-a645-11e4-95fd-c78c9ec356b6] - page.customHeaders:  - {}
[INFO  - 2015-01-27T16:58:05.366Z] Session [a9641420-a645-11e4-95fd-c78c9ec356b6] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"1.9.8","driverName":"ghostdriver","driverVersion":"1.1.0","platform":"linux-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"},"phantomjs.page.settings.userAgent":"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0"}
[INFO  - 2015-01-27T16:58:05.366Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: a9641420-a645-11e4-95fd-c78c9ec356b6
[ERROR - 2015-01-27T16:59:08.083Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1422377948079

The ERROR at the end is not part of my problem, but more of a bookend here. It occurs when the scraping algorithm looks for a link that's supposed to be on the page that shows up after logging in. Since we didn't successfully submit the login form, that link is naturally not present.


Solution

  • i'm not exactly certain which of my changes caused the breakthrough, but i can say i added the following code. edit: only two of these were new. removed one.

        webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.page.settings.localToRemoteUrlAccessEnabled"] = True
        webdriver.DesiredCapabilities.PHANTOMJS["phantomjs.page.settings.browserConnectionEnabled"] = True
    

    and i submitted the form itself, rather than clicking a button ( i had also tried this previously, prior to changing the above configs ).

            formEl = self.find_element_by_css_selector("form[name='login']")
            formEl.submit()