I am trying to use the TIdHTTP.Post() method to submit a form on a website that I just can't figure out. I have tried several iterations and changes to my code, and have hit a road block that I need to get help with. I am relatively new to TIdHTTP and its usage, so I beg forgiveness for anything that is just plain stupid about my code.
So far, I have been able to use the TIdHTTP.Get() method to obtain the HTML code from the site. Then I examine the <form> code within the HTML, and designed the below code to submit that form to the website.
I can't tell if my code isn't working because I am not using the Post() method correctly, or because I am not using the CookieManager correctly. All I receive is an "Internal Server Error" on execution.
Interestingly, the website to manually login requires you to enter your Account Number, Date Of Birth, and Password, but the form I see in the HTML only contains two variables for submission... Username (which is a combination of Acct and DOB, it seems) and submit. So I don't understand how/where it handles or posts the password variable?
Here is my current code in Delphi:
procedure TMSBS_App_GUI.SubmitClick(Sender: TObject);
Var
Response : String;
ResponseSet : TStringStream;
Params : TStringList;
IdHttp : TIDHttp;
IdSSL : TIdSSLIOHandlerSocketOpenSSL;
CookieMonster : TidCookieManager;
begin
Params := TStringList.Create;
Params.Add('username=' + 'username');
Params.Add('submit.value=' + 'submit');
idhttp := TIdhttp.Create;
idhttp.AllowCookies := True;
CookieMonster := TiDCookieManager.Create;
idHttp.CookieManager := CookieMonster;
idSSLOpenSSLHeaders.Load;
IdSSL := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
idHttp.ReadTimeout := 30000;
idHttp.IOHandler := idSSL;
idHttp.Get('https://' + website);
idhttp.Request.ContentType := 'application/x-www-form-urlencoded';
idhttp.Request.Referer := 'http://' + website;
idSSL.SSLOptions.Method := sslvTLSv1;
idSSL.SSLOptions.Mode := sslmUnassigned;
ResponseSet := TStringStream.Create(nil);
Try
Memo1.Text := idHttp.Post('https://' + website,Params);
Finally
Params.Free;
ResponseSet.Free;
End;
end;
This is the webpage:
<!-- SiteMinder Encoding=ISO-8859-1; -->
<!-- FCC File : (generic) caloglfn.fcc version 1.4-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
<link rel="stylesheet" type="text/css" href="styles.css" />
<!-- Cross-frame scripting prevention: This code will prevent this page from being encapsulated within HTML frames. Remove, or comment out, this code if the functionality that is contained in this SiteMinder page is to be included within HTML frames. -->
<SCRIPT type="text/javascript" src="https://ff.kis.v2.scr.kaspersky-labs.com/D0501246-9A02-314D-B50C-0C6D353C6332/main.js" charset="UTF-8"></script><link rel="stylesheet" crossorigin="anonymous" href="https://ff.kis.v2.scr.kaspersky-labs.com/2336C353D6C0-C05B-D413-20A9-6421050D/abn/main.css"/><script>
if (top !=self)
top.location=self.location;
</SCRIPT>
<title>Member/Pensioner Services Online Login</title>
<script>
function submit_form()
{
document.mos_form.username.value = document.mos_form.pUserID.value + document.mos_form.pDOB.value
document.mos_form.submit()
}
</script>
</head>
<body>
<div id="wrapper">
<div id="help">
<a style="border-bottom:none;" href="mso_pso_access_help.html" target="_blank"><img STYLE="border:none;" src="Help_button.png" alt="Help" align="right"> </a>
</div>
<div id="header">
<img class="crest" src="crest.png" alt="Crest - Superannuation Corporation" align="left">
<img class="logo" src="mso_pso.png" alt="Pensioner Services Online (PSO) and Member Services Online (MSO)" align="right">
</div>
<div id="toplinks">
</div>
<div id="form">
<form name="mos_form" method="POST" enctype="application/x-www-form-urlencoded" autocomplete="off">
<input type="hidden" name="autherrmsg" value="Login failed. Please try again."/>
<table width="100%" border="0" cellspacing="2" cellpadding="2">
<tr>
<td align="left">
<H1>
<span style="color: #3E842E;">Member Access</span>
</H1>
</td>
</tr>
<tr>
<td>
<p><font color="red"> </font></p>
<p>
To gain access to the complete range of online services, please enter your Membership Number, Date of Birth and Password below.
</p>
<p> If you need any help, click on Help in the top right-hand corner of this screen.</p>
<p> </p>
</td>
</tr>
</table>
<label for="hidden_username"></label>
<input type="hidden" name="username">
<label for="hidden_url"></label>
<input type="hidden" name="url" value="<ERROR_INFORMATION>" READONLY>
<label for="hidden_proxy"></label>
<input type="hidden" name="proxypath" value="<PROXY_PATH>">
<table width="900" border="0">
<tr height="38" valign="top">
<td width="200px" align="right" > <label for="mem_num" id="mem_num_label">Membership Number </label> </td>
<td width="200px" > <input id="pUserID" type="text" name="pUserID" value="" size="27" maxlength="13" id="mem_num"> </td>
<td width="500px">
<img STYLE="border:none;" src="mso_pso_question.png" onmouseover="this.style.cursor = 'help';"
title="Please enter your Membership Number.
This is either your Employment Number or Pension Reference Number as found on our correspondence to you.
If this is not available please Contact us.">
</td>
</tr>
<tr height="38" valign="top">
<td width="200px" align="right" > <label for="dob" id="dob_label">Date of Birth </label> </td>
<td width="200px" > <input type="text" name="pDOB" value="" size="27" maxlength="8" id="dob" placeholder="DDMMYYYY"> </td>
<td width="500px">
<img STYLE="border:none;" src="mso_pso_question.png" onmouseover="this.style.cursor = 'help';"
title="Please enter your date of birth in this format: DDMMYYYY (e.g. 01021955).">
</td>
</tr>
<tr height="38" valign="top">
<td width="200px" align="right" > <label for="acc_num" id="acc_num_label">Password</label> </td>
<td width="200px"> <input type="PASSWORD" name="password" value="" size="27" maxlength="30" id="acc_num"> </td>
<td width="500px">
<img STYLE="border:none;" src="mso_pso_question.png" onmouseover="this.style.cursor = 'help';"
title='Please enter your Password.
If you have forgotten your Password, use the "I've forgotten my password" link to reset your access credentials.
If you need to contact us, our details are available via the contact us button at the top right-hand corner of this screen.'>
</td>
</tr>
</tr>
<tr height="38" valign="top">
<td width="200px" width=200></td>
<td width="200px" align="right"> <a href="https://www.*****.au/no-scheme-provided/register-or-reset"> I've forgotten my password </a> </td>
<td width="500px">
</tr>
<tr height="38" valign="top">
<td colspan=2 align="right"><a href="https://www.*****.au/register-or-reset/no-scheme-provided"> Register</a>   <input type=button onclick=javascript:submit_form() value=Login> </td>
</tr>
<tr>
</table>
<script language="JavaScript">
<!-- the script here sets the focus on UserID field
document.mos_form.pUserID.focus();
document.mos_form.pUserID.select();
function enter(e)
{
if (navigator.appName == "Netscape")
whichASCII = e.which;
else
whichASCII = event.keyCode
if (whichASCII == 13 )
{
submit_form();
}
}
document.onkeypress = enter;
if (navigator.appName == "Netscape")
document.mos_form.pAccessCode.onkeypress = enter;
// End of script -->
</script>
<!-- SiteMinder Variables START -->
<input type=hidden name=target value="http://website">
<input type=hidden name=smauthreason value="0">
<input type=hidden name=smagentname value="boFynyFE9jczy7ra1lzqLmXPeVc9xLptAWQSI9ksks1Hx/oGQmJxQA7Fy25/Xt9X">
<!-- SiteMinder Variables END -->
</FORM>
</div>
<div id="footer">
<table width=100% border=0 cellspacing="0" cellpadding="0">
<tr>
<td height="30px" align="left" valign="middle" bgcolor="#3E842E">
<a href="http://www.*****.au/privacy" title="Privacy policy" target="_blank" class="wlink">Privacy</a> |
<a href="http://www.*****.au/disclaimer" title="Disclaimer" target="_blank" class="wlink">Disclaimer</a>
</td>
</tr>
<tr>
<td height="30px" colspan="2" valign="top" bgcolor="#949599" class="footer"><p class="footer"><span class="bold">Superannuation Company</span> ABN: ## ### ### ### AFSL: ###### RSEL: L#######<br /></td>
</tr>
</table>
</div>
</div>
<!--<script>
function getParameterByName(name, url) {
if (!url) url = window.location.href;
name = name.replace(/[[]]/g, "\$&");
var regex = new RegExp("[#&]" + name + "(=([^&#]*)|&|#|$)"),
results = regex.exec(url);
if (!results) return null;
if (!results[2]) return '';
return decodeURIComponent(results[2].replace(/+/g, " "));
}
document.getElementById('pUserID').value = getParameterByName('id');
</script> -->
</body>
</html>
And here is the Wireshark packet for the POST request:
Hypertext Transfer Protocol
POST /live/red_lojson/100eng.json?sh=0&ph=1383&ivh=928&dt=2720&pdt=214&ict=&pct=1&perf=widget%7C214%7C16%2Clojson%7C1027%7C656%2Csh%7C1031%7C0%2Csh%7C1035%7C16&rndr=render_toolbox%7C1375&cmenu=null&ppd=4&ppl=4&fbe=&xmv=0&xms=0&xmlc=0&jsfw=
Request Method: POST
Request URI [truncated]: /live/red_lojson/100eng.json?sh=0&ph=1383&ivh=928&dt=2720&pdt=214&ict=&pct=1&perf=widget%7C214%7C16%2Clojson%7C1027%7C656%2Csh%7C1031%7C0%2Csh%7C1035%7C16&rndr=render_toolbox%7C1375&cmenu=null&ppd=4&ppl=4&fbe=&xmv=
Request Version: HTTP/1.1
Host: m.addthis.com\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: en-US,en;q=0.5\r\n
Accept-Encoding: gzip, deflate\r\n
Referer: http://website/\r\n
Content-Length: 0\r\n
Content-Type: text/plain;charset=UTF-8\r\n
Cookie: na_tc=Y; uid=597293e6c72cb3be; na_id=2017072123300513069970337317; uvc=27%7C47%2C4%7C48%2C0%7C49%2C10%7C50%2C3%7C51; loc=MDAwMDBPQ0FVTlMxNDYxMzMxMjAwMDAwMDAwVg==; mus=0; ssc=pinterest%3B1%2Cgoogle%3B1\r\n
Connection: keep-alive\r\n
\r\n
The POST request you showed from Wireshark does not match the HTML you showed. In fact, that POST request is not even an HTML webform submission at all.
You don't need to create your own TIdCookieManager object, TIdHTTP can create one internally for you. And, for that matter, TIdHTTP can create the TIdSSLIOHandlerSocketOpenSSL object for you, too.
Also, you are leaking the CookieMonster and IdSSL objects (unless you are running this code on an ARC platform).
In any case, you are not filling in the TStringList correctly, not even close. You have to add an entry for every <input> field in the <form> that has a name and non-blank value. That includes all of the hidden fields, fields assigned by scripts, etc. Failing to do this can easily cause an "Internal Server Error" failure. You are providing a value for only 1 of the 10 input fields that the HTML form defines.
Based on the HTML you showed, try this instead:
procedure TMSBS_App_GUI.SubmitClick(Sender: TObject);
var
Params : TStringList;
IdHttp : TIdHTTP;
UserID, DOB, Password, AgentName, Response: String;
begin
IdSSLOpenSSL.LoadOpenSSLLibrary;
UserID := ...;
DOB := ...;
Password := ...;
IdHttp := TIdHTTP.Create;
try
IdHttp.AllowCookies := True;
IdHttp.ReadTimeout := 30000;
IdHttp.HandleRedirects := True;
//IdHttp.IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IdHttp);
Response := IdHttp.Get('https://' + website);
// I *suspect* the following value is randomly
// generated when the HTML is requested. If
// so, you will have to parse it out each time...
AgentName := 'boFynyFE9jczy7ra1lzqLmXPeVc9xLptAWQSI9ksks1Hx/oGQmJxQA7Fy25/Xt9X';
Params := TStringList.Create;
try
Params.Add('autherrmsg=Login failed. Please try again.');
Params.Add('username=' + UserID + DOB);
Params.Add('url=<ERROR_INFORMATION>');
Params.Add('proxypath=<PROXY_PATH>');
Params.Add('pUserID=' + UserID);
Params.Add('pDOB=' + DOB);
Params.Add('password=' + Password);
Params.Add('target=http://website');
Params.Add('smauthreason=0');
Params.Add('smagentname=' + AgentName);
IdHttp.Request.Referer := 'http://' + website;
Response := IdHttp.Post('https://' + website, Params);
Memo1.Text := Response;
finally
Params.Free;
end;
finally
IdHttp.Free;
end;
end;