How to make two POST Requests in one session (JSESSIONID)/cookie

Question

Okay so this is a complicated question. It's been over 2 and half days of research and debugging and I'm currently here.

Goal: Fetch data from a government GST/HST lookup website. Since it is old (uses as Java applet) and does not have REST API support, I have to manually do 2 web requests to the website. (After an entire day of debugging I found that I have to accept the terms and conds first before starting a search since that acceptance is stored in a cookie. The link the website is here: Government Search Website)

Problem Encountered: Flowgear's web request node does not allow you to specify a 'Cookie' header so I can't tell it to use a specific cookie on two separate requests. So I had to do this:

Note: I'm trying to use the same Cookie on two POST requests so I get proper data back for that specific session.

Current Problem: I get this error: "Script execution error: Cannot send a content-body with this verb-type."

Code:

using System;
using System.IO;
using System.Collections.Generic;
using System.Xml;
using System.Linq;
using System.Xml.Linq;
using System.Net;
using System.Text;

namespace CSharpScript
{
  public partial class Processor
  {  
    public object Process()
    {
        var request = (HttpWebRequest)WebRequest.Create("https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryservlet");
        var postData = "iagree=yes";
        var data = Encoding.ASCII.GetBytes(postData);

        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.ContentLength = data.Length;
        request.KeepAlive = true;
        request.Host  = "www.businessregistration-inscriptionentreprise.gc.ca";
        Uri target = new Uri("https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryservlet");
        CookieContainer reqCookies = new CookieContainer();
        request.CookieContainer = reqCookies;
        request.CookieContainer.Add(new Cookie("id","YxmlT6nC0FPUakMqOF7S5ZGM-URCkTR-sjddOY02Gz2XfOodrOia!1465666898"){ Domain = target.Host });
        //request.Headers["Cookie"] = "YxmlT6nC0FPUakMqOF7S5ZGM-URCkTR-sjddOY02Gz2XfOodrOia!1465666898";
        using (var stream = request.GetRequestStream())
        {
          stream.Write(data, 0, data.Length);
         }

        var response = (HttpWebResponse)request.GetResponse();
        var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
        //Part 2 
        var request2 = (HttpWebRequest)WebRequest.Create("https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryPromptSubmit.do");
        var postData2 = "businessName=COMPANYNAMEASSTRINGGOESHERE";
        postData2 += "&businessNumber=SOMENUMBERGOESHERE";
        postData2 += "&requestDate=2014-09-02";
        var data2 = Encoding.ASCII.GetBytes(postData2);

        request2.Method = "POST";
        request2.ContentType = "application/x-www-form-urlencoded";
        request2.ContentLength = data2.Length;
        request2.KeepAlive = true;
        request2.Host  = "www.businessregistration-inscriptionentreprise.gc.ca";
        //Uri target = new Uri("https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryservlet");
        request2.CookieContainer = reqCookies;
        //request.Headers["Cookie"] = "YxmlT6nC0FPUakMqOF7S5ZGM-URCkTR-sjddOY02Gz2XfOodrOia!1465666898";
        using (var stream = request.GetRequestStream())
        {
          stream.Write(data2, 0, data2.Length);
        }

        var response2 = (HttpWebResponse)request2.GetResponse();

        responseString = new StreamReader(response2.GetResponseStream()).ReadToEnd();
        return responseString;
    }
}

}

Debugging: If i remove everything under 'Part 2' except the return statement, the first POST works fine. Before this, I had Part 2 in its own script node and it returned data successfully, but the search results were missing. (Since the session info/cookie info wasn't carried over)

Important Note: I have done these exact two requests on Postman and it works successfully there.

Any idea on how to submit a post request with data to this website through flowgear would be greatly appreciated. :(

UPDATE: For the second POST request I realized I forgot to change

using (var stream = request.GetRequestStream())

to match the response object 'request2':

using (var stream = request2.GetRequestStream())

The web requests work successfully now! Solved!

However, my question still remains: How do I make two separate requests share a cookie or session and how do I set the Connection header property to 'Keep-alive'?

Answer 1

Try creating a connection for the Web Request and providing a value for the session key property. Then use the same connection on the two requests which causes them to share the same cookie jar. You won't need to manually try to inject headers.

Note that the cookie jar is scoped to the Flowgear site and is in-memory so you can have the two requests in different workflows. Eg. One in a parent, one in a sub workflow.