page posting issue when working in Screen Scraping

Question

I am working on screen scraping and done successfully in 3 websites, I have an issue in last website

here is my url, When I hit with my parameter, it is showing result on next page, simply posting to other page and showing the result fine on other page

Here is My Test

However, when I hit from my application, since here I don't have an option to post, it only fetch html of requested page that is obviously my above mention HTML test link, that actually have parameter in URL to get the result.

How can I handle this situtation? Please give me hint.

Thanks

here is my C# code, I am using HTMLAgality

String url;
HtmlWeb hw = new HtmlWeb();
HtmlDocument doc;
url = "http://mysampleURL";
doc = hw.Load(url);

Answer 1

Use the WebClient class for posting the form of the first page with the expected input values. The input values can be found in the source of the first page, but it's also possible to capture them using Fiddler which is imho a great tool for these scenarios.

Example:

NameValueCollection values = new NameValueCollection();
values.Add("action","hotelPackageWizard@searchHotelOnly");
values.Add("packageType","HOTEL_ONLY");
// etc..
WebClient webclient = new WebClient();
webclient.Headers.Add("Content-Type","application/x-www-form-urlencoded");
byte[] responseArray = webclient.UploadValues("http://www.expedia.com/Hotels?rfrr=-905&","POST", values);
string response = System.Text.Encoding.ASCII.GetString(responseArray);