httpclient delivering different results

Question

I am having issues interfacing with a PowerSchool system using Apache HttpClient 4.5.6. I have left out methods that aren't necessary for this question, but I am 100% sure that they work as intended.

When using Postman or requests.py, using the exact same form data (I've verified it across python, java, and Firefox), I get the intended HTML page with the desired grades and class information in between a <!-- start student content --> and <!-- end student content --> comment (over 1000 lines). However, when using the same form data and headers in Java, the resulting HTML page contains only this:

<!-- start student content -->
<div id="quickLookup">

<tr>
<th class="right" colspan="19">Attendance Totals</th>
<th>0</th>
<th>0</th>
</table>
<table border="0" cellpadding="3" cellspacing="1" width="100%">


<tr>
<td align="center">Current Cumulative GPA (Q1): X.XXXX</td>
</tr>

<tr>
<td align="center"><a href="home.html?schoolid=XXXX&showdropped=true&91146885685933636948">Show dropped classes also</a></td>
</tr>
</table>

<tr>
<th class="right" colspan="10">Attendance Totals</th>
<th>0</th>
<th>0</th>
<th>0</th>
<th>0</th>
</table>

</div>
<!-- end student content -->

The disconnect between the two different results makes no sense to me, since as far as I know Postman and requests.py don't execute javascript. I would expect HttpClient's result to be the same. Here is my code:

private static final BasicCookieStore cookieStore = new BasicCookieStore();
private static final HttpClient client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();

public static void main(String[] args) {

        String baseUrl = "https://powerschoolinstallurl/";
        String username = "username";
        String password = "password";

        try {
            // get hidden data fields, calc hmac data
            HashMap<String, String> result = getAuthCodes(baseUrl);
            String dbpwField = getDBPWField(result.get("contextData"), password);
            String pwField = getPWField(result.get("contextData"), password);

            List<NameValuePair> form = new ArrayList<>();
            form.add(new BasicNameValuePair("pstoken", result.get("pstoken")));
            form.add(new BasicNameValuePair("contextData", result.get("pstoken")));
            form.add(new BasicNameValuePair("dbpw", dbpwField));
            form.add(new BasicNameValuePair("serviceName", "PS Parent Portal"));
            form.add(new BasicNameValuePair("pcasServerURL", "/"));
            form.add(new BasicNameValuePair("credentialType", "User Id and Password Credential"));
            form.add(new BasicNameValuePair("account", username));
            form.add(new BasicNameValuePair("pw", pwField));
            form.add(new BasicNameValuePair("ldappassword", password));

            UrlEncodedFormEntity requestEntity = new UrlEncodedFormEntity(form);

            HttpPost postMethod = new HttpPost(baseUrl + "guardian/home.html");
            postMethod.setEntity(requestEntity);

            HttpResponse rawResponse = client.execute(postMethod);
                    System.out.println(rawResponse.getStatusLine().getStatusCode());
            try {
                String responseString = new BasicResponseHandler().handleResponse(rawResponse);
                System.out.println(responseString);
            } catch (HttpResponseException ignore) {}
            System.out.println(cookieStore.getCookies().toString());

            HttpGet getMethod = new HttpGet(baseUrl + "guardian/home.html");
            // replicating headers, result is the same nontheless
            getMethod.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
            getMethod.setHeader("Accept-Encoding", "gzip, deflate, br");
            getMethod.setHeader("Accept-Language", "en-US,en;q=0.5");
            getMethod.setHeader("Referer", "https://powerschoolinstallurl/public/home.html");
            getMethod.setHeader("DNT", "1");
            getMethod.setHeader("Connection", "keep-alive");
            getMethod.setHeader("Cache-Control", "no-cache");
            getMethod.setHeader("Host", "ps.install.domain");
            getMethod.setHeader("Upgrade-Insecure-Requests", "1");
            getMethod.setHeader("Pragma", "no-cache");

            HttpResponse resp2 = client.execute(getMethod);
            String responseString2 = new BasicResponseHandler().handleResponse(resp2);
            System.out.println(responseString2);


        } catch (IOException e) {
            e.printStackTrace();
        }
}

Note: there is no authentication issue whatsoever, the page is returned normally besides that one detail.

Answer 1

I ended up using Jsoup, following the exact same logic and got my desired result. Still no clue why HttpClient didn't work.