HttpWebRequest: How to find a postal code at Canada Post through a WebRequest with x-www-form-enclosed?

Question

I'm currently writing some tests so that I may improve my skills with the Internet interaction through Windows Forms. One of those tests is to find a postal code which should be returned by Canada Post website.

1. My default URL setting is set to: http://www.canadapost.ca/cpotools/apps/fpc/personal/findByCity?execution=e4s1
2. The required form fields are: streetNumber, streetName, city, province
3. The contentType is "application/x-www-form-enclosed"

EDIT: Please consider the value "application/x-www-form-encoded" instead of point 3 value as the contentType. (Thanks EricLaw-MSFT!)

The result I get is not the result expected. I get the HTML source code of the page where I could manually enter the information to find the postal code, but not the HTML source code with the found postal code. Any idea of what I'm doing wrong?

Shall I consider going the XML way? Is it first of all possible to search on Canada Post anonymously?

Here's a code sample for better description:

public static string FindPostalCode(ICanadadianAddress address) {
   var postData = string.Concat(string.Format("&streetNumber={0}", address.StreetNumber)
    , string.Format("&streetName={0}", address.StreetName)
    , string.Format("&city={0}", address.City)
    , string.Format("&province={0}", address.Province));

   var encoding = new ASCIIEncoding();
   byte[] postDataBytes = encoding.GetBytes(postData);
   request = (HttpWebRequest)WebRequest.Create(DefaultUrlSettings);
   request.ImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Anonymous;
   request.Container = new CookieContainer();
   request.Timeout = 10000;
   request.ContentType = contentType;
   request.ContentLength = postDataBytes.LongLength;
   request.Method = @"post";
   var senderStream = new StreamWriter(request.GetRequestStream());
   senderStream.Write(postDataBytes, 0, postDataBytes.Length);
   senderStream.Close();
   string htmlResponse = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

   return processedResult(htmlResponse); // Processing the HTML source code parsing, etc.
}

I seem stuck in a bottle neck in my point of view. I find no way out to the desired result.

EDIT: There seems to have to parameters as for the ContentType of this site. Let me explain.

• There's one with the "meta"-variables which stipulates the following:

meta http-equiv="Content-Type" content="application/xhtml+xml, text/xml, text/html; charset=utf-8"

• And another one later down the code that is read as:

form id="fpcByAdvancedSearch:fpcSearch" name="fpcByAdvancedSearch:fpcSearch" method="post" action="/cpotools/apps/fpc/personal/findByCity?execution=e1s1" enctype="application/x-www-form-urlencoded"

My question is the following: With which one do I have to stick?

Let me guess, the first ContentType is to be considered as the second is only for another request to a function or so when the data is posted?

EDIT: As per request, the closer to the solution I am is listed under this question: WebRequest: How to find a postal code using a WebRequest against this ContentType=”application/xhtml+xml, text/xml, text/html; charset=utf-8”?

Thanks for any help! :-)

Answer 1

I'm trying to see a reason why you are not using the WebClient class:-

var fields = new NameValueCollection();
fields.Add("streetnumber", address.StreetNumber);
fields.Add("streetname", address.StreetName);
fields.Add("city", address.City);
fields.Add("province", address.Province);

var wc = new WebClient();
byte[] resultData = wc.UploadValues(url, fields);
string result = Encoding.Default.GetString(resultData);

You might want to check the encoding used by the server when sending the results, if it uses UTF-8 change the last line to:-

string result = Encoding.UTF8.GetString(resultData);

Some issues I spot in your orginal code:-

1. The first field is prefixed with &, that shouldn't be there.
2. You need call use Uri.EscapeDataString on each field value.
3. You are attempting to construct a memory stream around the result of GetRequestStream, I can't see what that would achieve even if MemoryStream had such a constructor but it doesn't anyway. Just write directly to the stream returned by GetRequestStream

If you have already done so get yourself a copy of fiddler so you can observe what occurs when a standard form requests the data sucessfully and what your code is doing.

Edit: If you have evidence that the lack of a cookie container is what causes WebClient not to work then you could try this approach:-

public class MyWebClient : WebClient
{

    protected override WebRequest GetWebRequest (Uri address)
    {
      WebRequest request = (WebRequest) base.GetWebRequest (address);

      request.Container = new CookieContainer();
      return request;
    }
}

Now use my code above to but instead od instancing WebClient instance MyWebClient instead.