I am having some trouble using Requests to retrieve some data from a webpage. It uses Siteminder and the initial form only has three fields but when I submit it, my password is changed to hex and other fields are added. Can't seem to get it to work at all. I keeps returning an error message.
Any help is appreciated and I apologize for the long post!
Edit: included the two JavaScript functions because they change the data.
Python:
from requests import session
with session() as s:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
payload = {
'USER': 'username',
'PASSWORD': 'pw',
'TARGET': 'https://www.THISSITE.com/pg'
}
resp = s.post('https://www.THISSITE.com/THIS/fcc/THISSITE.fcc', headers=headers, data=payload )
html = resp.text
print(html)
Form:
<form
id="login"
method="post"
name="Login"
action="https://www.THISSITE.com/THIS/fcc/THISSITE.fcc">
<input
type="hidden"
name="TARGET"
value="https://www.THISSITE.com/pg"
></input>
<div class="form-group">
<input
type="text"
id="USER"
name="USER"
value=""
></input>
<div class="form-group">
<input
type="password"
id="PASSWORD"
name="PASSWORD"
value=""
></input>
</div>
<input
type="submit"
name="OK"
value="Login"
onclick="submitAuthForm(this.form);"
></input>
submitAuthForm(form):
function submitAuthForm(form) {
var strval = form.PASSWORD.value;
if(!isJson(strval)){
var info = {};
info["Password"] = hexEncode(strval);
form.PASSWORD.value = JSON.stringify(info);
}
}
hexEncode(str):
function hexEncode(s){
var chrsz = 8;
var hexcase = 0;
function str2binb (str) {
var bin = Array();
var mask = (1 << chrsz) - 1;
for(var i = 0; i < str.length * chrsz; i += chrsz) {
bin[i>>5] |= (str.charCodeAt(i / chrsz) & mask) << (24 - i%32);
}
return bin;
}
function Utf8Encode(string) {
string = string.replace(/\r\n/g,"\n");
var utftext = "";
for (var n = 0; n < string.length; n++) {
var c = string.charCodeAt(n);
if (c < 128) {
utftext += String.fromCharCode(c);
}
else if((c > 127) && (c < 2048)) {
utftext += String.fromCharCode((c >> 6) | 192);
utftext += String.fromCharCode((c & 63) | 128);
}
else {
utftext += String.fromCharCode((c >> 12) | 224);
utftext += String.fromCharCode(((c >> 6) & 63) | 128);
utftext += String.fromCharCode((c & 63) | 128);
}
}
return utftext;
}
function binb2hex (binarray) {
var hex_tab = hexcase ? "0123456789ABCDEF" : "0123456789abcdef";
var str = "";
for(var i = 0; i < binarray.length * 4; i++) {
str += hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8+4)) & 0xF) +
hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8 )) & 0xF);
}
return str;
}
s = Utf8Encode(s);
return binb2hex(str2binb(s));
}
Parameters when I submit via webpage:
SMENC: UTF-8
SMLOCALE: US-EN
target: https://www.THISSITE.com/pg
smauthreason: 27
smagentname: mR2h1e4BPUPZ5eTpyZckvJXpXO1mE5RpNTYtnh9C8sMfqiHlbrnBjW2SNjbwIRz+
type:
realmoid:
smusermsg:
USER: username
PASSWORD: {"TokenId":"longstringoflettersandnumbersHEX???","Password":""}
The hexEncode function is taking a string and converting to a series of hex representations of the constituent bytes of it's UTF8 encoded representataion. The equivalent in Python would be to encode an input unicode string in UTF-8 and then re-encode the result of that in hex, e.g.
>>> import binascii
>>> binascii.hexlify('d'.encode('utf-8'))
b'64'
>>> binascii.hexlify('¡¢£¤¥'.encode('utf-8'))
b'c2a1c2a2c2a3c2a4c2a5'
Note: in Python 2.7 this would be —
>>> 'd'.encode('utf-8').encode('hex')
'64'
>>> u'¡¢£¤¥'.encode('utf-8').encode('hex')
'c2a1c2a2c2a3c2a4c2a5'
If you test that with your example password, it should produce the same output as for the website, with one caveat.
hexEncode('d')
"64000000"
Notice that the Javascript adds a number of trailing 0's, making the length of the string a multiple of 8. We need to pad the result we have to get the same output.
>>> s = binascii.hexlify('d'.encode('utf-8'))
>>> n = len(s)
>>> from math import ceil
>>> next_8_multiple = int(ceil(n/8.0) * 8)
>>> s.ljust(next_8_multiple, b'0')
b'6400000000'
We can wrap that up in a complete function:
from math import ceil
import binascii
def hex_encode_and_pad(s):
hex = binascii.hexlify(s.encode('utf-8'))
n = len(hex)
next_8_multiple = int(ceil(n/8.0) * 8)
zeros_to_append = next_8_multiple - n
return hex.ljust(next_8_multiple, b'0')
This now gives the same result as for the Javascript function:
>>> hex_encode_and_pad('d')
'64000000'
The next step would be to wrap it in a string representation of the JSON. You can either do that by hand-coding the string + just interpolating the token, e.g.
value = '{"TokenId":"%s","Password":""}' % token
Or by creating the JSON string from a Python dictionary —
import json
data = {'TokenId': token, 'Password': ''}
value = json.dumps(data)
The complete code based on the example request shown above would be:
import binascii
import json
from math import ceil
from requests import session
def hex_encode_and_pad(s):
hex = binascii.hexlify(s.encode('utf-8'))
n = len(hex)
next_8_multiple = int(ceil(n/8.0) * 8)
zeros_to_append = next_8_multiple - n
return hex.ljust(next_8_multiple, b'0')
with session() as s:
password = u'your_password'
token = hex_encode_and_pad(password)
data = {'TokenId': token, 'Password': ''}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
payload = {
'USER': 'username',
'PASSWORD': json.dumps(data),
'TARGET': 'https://www.THISSITE.com/pg'
}
resp = s.post('https://www.THISSITE.com/THIS/fcc/THISSITE.fcc', headers=headers, data=payload )
html = resp.text
print(html)