Search code examples
javajsouphelpshift

Jsoup login with form (post)

After reading some examples, I wanted to implement a crawler for helpshift with login such as:

https://target.helpshift.com/login/?next=%2Fadmin%2Fissues%2F

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class JsouptTest {

    public static void main(String[] args) throws Exception {
        int x = 1;
        Connection.Response loginForm = Jsoup.connect("https://target.helpshift.com/login/?next=%2Fadmin%2Fissues%2F" + x + "%2F")
                .method(Connection.Method.GET)
                .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0")
                .execute();

        Document document = Jsoup.connect("https://target.helpshift.com/login/")
                .data("cookieexists", "false")
                .data("username", "[email protected]")
                .data("password", "123456")
                .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0")
                .cookies(loginForm.cookies())
                .post();
        System.out.println(document);

    }

}

However, I am getting this error:

Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=https://target.helpshift.com/login/ at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:537) at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:493) at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:205) at org.jsoup.helper.HttpConnection.post(HttpConnection.java:200) at edu.utfpr.helpcrawler.JsouptTest.main(JsouptTest.java:32)

Solution

  • If you check the request headers you will see the it sends the cookies as you've done, but it includes a part of the cookie in the form data too. Add this to your second request

    .data("_csrf_token", loginForm.cookie("_csrf_token"))