Search code examples
perlsessioncookieslwp

Create a session with necessary cookies using Perl

I will try to give as many details as possible.

Using Perl, I am trying to download the list of Impact Factors of all Journals for the JCR (Journal Citation Reports) from the Web of Knowledge (WoS) (url: http://admin-apps.webofknowledge.com/JCR/JCR?RQ=SELECT_ALL&cursor=1). It should be noted that the link is only accessible using IPs from most academic institutions, like mine, or from paying users.

Whenever I visit the link (using the browser) and sometime has passed, the session has expired so I need to click on a link (http://admin-router.webofknowledge.com/?DestApp=JCR) to establish a new session. Using my rudimentary knowledge of Firebug I found that when I am authenticated I get a cookie named jcrsid with a random string value.

My question is the following: Using Perl's LWP module, what steps should I take in order to establish a session and download the web pages I need. (If I use LWP to download the page, i download exactly the same page that appears when I am asked to establish a new session using the browser).

Solution

  • You need to set a cookie_jar on your LWP::UserAgent object. LWP::UA will then store all the cookies it encounters in a HTTP::Cookies object and handle them for you. They'll be sent with every request.

    You can use an empty hash reference {} to store your cookie in memory. There is no need to use a file for this kind of job.

    use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
    cookie_jar => {},
);

    That's it. Then you would to get the link that gives you the login cookie first.

    my $res_login = $ua->get('http://admin-router.webofknowledge.com/?DestApp=JCR');
die $res_login->status_line unless $res_login->is_success;

    Afterwards you can go about loading your pages. The cookie will be sent automatically.