curlpostsingle-sign-onsession-cookies
Can not sign on to usgs.gov using curl to store session-cookies
I am trying to sign on to ers.cr.usgs.gov using curl. I have been trying lots of different methods over the past couple days, yet I can never seem to get the second part of session cookies sent by the response. I've tried directly saving the post through Firefox developer tools as a curl request, and replacing the csrf_token and __ncforminfo with the proper values, and still nothing. I will include the curl command, along with the log output, and some screenshots of the successful sign-on, below.
Any help would be immensely appreciated. Thank you.
- First, I am using curl to download the page using a
GETrequest, which I parse to get the hidden fields for the form
curl -o temp.html "$LOGIN"
TOKEN=$(sed -n 's/.*name="csrf_token"\s\+value="\([^"]\+\).*/\1/p' temp.html)
ENCODED_TOKEN=$(urlencode $TOKEN)
FORMINFO=$(sed -n 's/.*name="__ncforminfo"\s\+value="\([^"]\+\).*/\1/p' temp.html)
ENCODED_FORMINFO=$(urlencode $FORMINFO)
- Next, I take these values and send off a curl post request
curl --verbose \
--cookie-jar cookies.txt \
-H "Host: ers.cr.usgs.gov" \
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "Origin: https://ers.cr.usgs.gov" \
-H "Connection: keep-alive" \
-H "Referer: https://ers.cr.usgs.gov/login/" \
-H "Upgrade-Insecure-Requests: 1" \
--data "username=$USERNAME&password=$PASSWORD&csrf_token=$ENCODED_TOKEN&__ncforminfo=$ENCODED_FORMINFO" https://ers.cr.usgs.gov/login/
- This is the verbose output of curl
* Trying 2001:49c8:4000:122c::7...
* TCP_NODELAY set
* Trying 152.61.136.7...
* TCP_NODELAY set
* Connected to ers.cr.usgs.gov (152.61.136.7) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Virginia; L=Reston; O=U.S. Geological Survey; OU=USGS; CN=*.cr.usgs.gov
* start date: Apr 5 00:00:00 2019 GMT
* expire date: Jun 10 12:00:00 2020 GMT
* subjectAltName: host "ers.cr.usgs.gov" matched cert's "*.cr.usgs.gov"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> POST /login/ HTTP/1.1
> Host: ers.cr.usgs.gov
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate, br
> Content-Type: application/x-www-form-urlencoded
> Origin: https://ers.cr.usgs.gov
> Connection: keep-alive
> Referer: https://ers.cr.usgs.gov/login/
> Upgrade-Insecure-Requests: 1
> Content-Length: 196
>
* upload completely sent off: 196 out of 196 bytes
< HTTP/1.1 403 Forbidden
< Date: Tue, 05 Nov 2019 15:41:14 GMT
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=31536000; includeSubDomains
* cookie size: name/val 9 + 26 bytes
* cookie size: name/val 4 + 1 bytes
* cookie size: name/val 6 + 0 bytes
* cookie size: name/val 8 + 0 bytes
* Added cookie PHPSESSID="8rd75nf2ugv0ono7djbh29v30l" for domain ers.cr.usgs.gov, path /, expire 0
< Set-Cookie: PHPSESSID=8rd75nf2ugv0ono7djbh29v30l; path=/; secure; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
* cookie size: name/val 9 + 26 bytes
* cookie size: name/val 4 + 1 bytes
* cookie size: name/val 6 + 0 bytes
* cookie size: name/val 8 + 0 bytes
* Added cookie PHPSESSID="8rd75nf2ugv0ono7djbh29v30l" for domain ers.cr.usgs.gov, path /, expire 0
< Set-Cookie: PHPSESSID=8rd75nf2ugv0ono7djbh29v30l; path=/; secure; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
* cookie size: name/val 9 + 26 bytes
* cookie size: name/val 4 + 1 bytes
* cookie size: name/val 6 + 0 bytes
* cookie size: name/val 8 + 0 bytes
* Replaced cookie PHPSESSID="54vnism9of8r7n9da2pqn8rjla" for domain ers.cr.usgs.gov, path /, expire 0
< Set-Cookie: PHPSESSID=54vnism9of8r7n9da2pqn8rjla; path=/; secure; HttpOnly
< Content-Length: 0
< Keep-Alive: timeout=15, max=50
< Connection: Keep-Alive
< Content-Type: text/html; charset=UTF-8
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host ers.cr.usgs.gov left intact
- and this is the cookies.txt (as you can see it lacks the
EROS_SSO_production_securecookie that should be returned by the response)
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_ers.cr.usgs.gov FALSE / TRUE 0 PHPSESSID blahblahblahhereissessionstuff
- Here is the post info for a successful sign-on from the Firefox Developer tools
Solution
I found my own answer on this one. When I first pull the page to get the csrf_token and __ncforminfo, the page is setting a PHPSESSID cookie that also needs to be present for the SSO to send back the correct information. Changing my two curl requests as below, allowed the POST to succeed.
curl --cookie-jar cookies.txt -o temp.html "$LOGIN"
and
curl \
--cookie cookies.txt \
--cookie-jar cookies.txt \
...
--data "username=$USERNAME&password=$PASSWORD&csrf_token=$ENCODED_TOKEN&__ncforminfo=$ENCODED_FORMINFO" https://ers.cr.usgs.gov/login/
- When did C allow casting constants and variables to struct for assignment?
- Why in C can I initialize more values than the size of an array?
- Using c to create a file and write+read to it, what went wrong?
- Does write system call need the data to be read to give >-1 return?
- Difference between fgetc() and read() function in C
- How can I get to know the IP address for interfaces in C?
- read fails with EFAULT
- In this case, how to achieve modularity and information hiding at the same time?
- Are global variables always initialized to zero in C?
- OpenGL function calls seemingly affecting unrelated data
- What Values of Variables x and y Will Produce Incorrect Results When Testing for Overflow After Subtraction of x-y in the Following Program?
- How to write a general-type queue library in c?
- Which gcc and g++ version support which standard of c and c++?
- How to dereference a member in a struct whose definition is not visible?
- How to name a type in a meaningful way?
- What are Vectors and < > in C?
- Why does 1.0/100.0 == 0.1/10.0 give True?
- How to organize the receive msg and user current input in C network such that it's clean
- Using inclusive scan syntax in OpenMP in the C language
- Unable to understand context in book "OOP in C" by Axel Schreiner
- variable-length array in struct with TI compiler in C (socket programming)
- A faster way to test 32bpp DDBs for a valid Alpha channel
- How can I compile the zephyr example-application as a freestanding application?
- Why does my forked process sometimes overwrite data in a file?
- _Generic in C needs typecasting?
- Declaring/defining an unused variable changes the output from an unrelated variable
- How to use gdb to explore the stack/heap?
- How can I read an input string of unknown length?
- Confused by difference between expression inside if and expression outside if
- Fast Arc Cos algorithm?