Not able to POST ASP-NET form with WWW::Mechanize or LWP::UserAgent

Question

I need to automate the extraction of a products catalog, for which we are authorized distributors. The company is Ingram Micro and they're huge, but they don't provide a simple way (like a web service) to query their catalog and stock, so we've been extracting it with LWP::UserAgent so far, daily, for our online store database.

Recently, they changed their website, and there's something that I just can't detect preventing my Perl script from doing a succesful login. If I try directly on the browser, I login succesfully, but If I try submiting the form from Perl, I get the login page again (like an unsuccessful login).

Their site is made on AspNet and full of hidden form fields generated by ASP. At first, I was doing my own 'spider' to get the login page, read all the hidden field values and include them in my POST login request along with the username and password, but now, after this problem, I started using WWW::Mechanize to do the requests and have that task automated correctly.

However, not WWW::Mechanize or LWP::UserAgent are able to do a succesful login.

I've even tried looking at the exact HTTP conversation between the browser and the server, and including the same exact headers on my automated request with the correct POST content, and even doing that I get an unsuccesful login from Perl and a succesful login from the browser.

I'm really desperate, I just cannot detect the problem and we are not able to run the store without this catalog update.

You can see the login page here: https://www.imstores.com/ingrammicromx/

Please, if someone has experience with a similar issue or someone can look at that page and detect what makes a browser different from a WWW::Mechanize request, please let me know.

Thank you.

Answer 1

Usually I use this way to test what's wrong:

1. Clear all cookies in your browser
2. Disable Javascript
3. Open login page
4. Try to login

If you have success login than you can do this with WWW::Mechanize without additional tricks, just don't forget to use button => "submit_button_name" like this:

$mech->submit_form(
    with_fields => {
        login_field_field => $login,
        password_field_name => $password,
    },
    button => "submit_button_name",
);

Even if you can't login from the browser with Javascript disabled you still can do this with Mechanize but you need to look at the browser's HTTP request (I use HTTPFox for this) and make exactly the same request (including headers) from Mechanize.