Accessing JSON in using JSESSION ID or Authentication

Question

I've created a simple Java project (not a Dynamic Web Project) in Eclipse.

I'd like to be able to access a web URL that returns some JSON values.

I'm able to get to the URL using HttpClient and all that but I'm unable to Authenticate my session, which is required for me to get access to the JSON data. If you're not authenticated, the URL redirects you to a login page.

I've tried numerous ways of authenticating my session but none seem to work -- I always end up up getting login page's HTML data instead of the JSON.

I've also noticed that a JSESSION ID is passed into the Request Header but I can't seem to figure out how to create a JSESSION ID and how to pass that into the GET request.

So basically, I can solve this issue in two ways:

1. Authenticate my session and then access the URL to get the JSON
2. Create a JSESSION ID and pass that into the GET request to get the JSON

Anyone know how to do either?

Eventually, I'd like to use the data I collected from the JSON to do some Selenium testing so do I need to do all this within a Dynamic Web Project (in Eclipse) or can it be all done within a regular Java project?

---

Attempt 1

URL url = new URL(_url);

InputStream ins = url.openConnection().getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(ins));
StringBuilder builder = new StringBuilder();

for (String line = null; (line = reader.readLine()) != null;) {
    builder.append(line).append("\n");
}

System.out.println("RESPONSE:\n\n" + builder);

JSONTokener tokener = new JSONTokener(builder.toString());
JSONArray finalResult = new JSONArray(tokener);

System.out.println("JSON:\n\n" + finalResult.toString());

Attempt 1 Results

I end up printing out the HTML code for the log-in page and then get the following error message since there is no JSON on the page:

Exception in thread "main" org.json.JSONException: A JSONArray text must start with '[' at character 1

---

Attempt 2

I have also tried the code found in Oracle's Http Authentication page:

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URL;

public class RunHttpSpnego {

    static final String kuser = "username"; // your account name
    static final String kpass = "password"; // your password for the account

    static class MyAuthenticator extends Authenticator {
        public PasswordAuthentication getPasswordAuthentication() {
            // I haven't checked getRequestingScheme() here, since for NTLM
            // and Negotiate, the usrname and password are all the same.
            System.err.println("Feeding username and password for " + getRequestingScheme());
            return (new PasswordAuthentication(kuser, kpass.toCharArray()));
        }
    }

    public static void main(String[] args) throws Exception {
        Authenticator.setDefault(new MyAuthenticator());
        URL url = new URL(args[0]);
        InputStream ins = url.openConnection().getInputStream();
        BufferedReader reader = new BufferedReader(new InputStreamReader(ins));
        String str;
        while((str = reader.readLine()) != null)
            System.out.println(str);
    }
}

Answer 1

When you log in, the server sets the session ID as a cookie in your browser using a Set-Cookie header in the response.

Set-Cookie: name=value
Set-Cookie: name2=value2; Expires=Wed, 09-Jun-2021 10:18:14 GMT

The browser then provides this cookie to the server with each subsequent request, in the request header.

Cookie: name=value; name2=value2

HttpClient can do this administration for you, just like the browser. See http://hc.apache.org/httpclient-3.x/cookies.html So then you'd first tell HttpClient to surf to the login page, it'll fetch your cookie and on subsequent requests it'll send it along.

You can also take matters into your own hands and set the cookie manually. In HttpClient 3 this looks like this:

HttpMethod method = new GetMethod();
method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES);
method.setRequestHeader("Cookie", "special-cookie=value");

How to set the cookie in HttpClient 4 can be found here: Apache HttpClient 4.0.3 - how do I set cookie with sessionID for POST request

Or you can take it a little bit lower level and add the full header.

HttpGet httpget = new HttpGet(url); 
httpget.addHeader("Cookie", "JSESSIONID=...");