C# Post data to form

Question

I previously used Selenium in order to automate web processes. However, it doesn't always work reliable and more importantly, it is dependent from the screen size which is not necessarily always the same.

Now I have a form like this (very simplified):

<form type="POST" name="demoform">
  First name:<br>
  <input type="text" name="firstname"><br>
  Last name:<br>
  <input type="text" name="lastname">
</form>

Previously, I would have searched for the the inputs names and enter the data. But now I want to do this using a different way. I wonder how to send data via POST and how to get the entire HTML result of the web page. Further I wonder if the latter is just a HTML response or if it's the the page's HTML that actual users would see.

Answer 1

I previously used Selenium in order to automate web processes. However, it doesn't always work reliable and more importantly, it is dependent from the screen size

You haven't really backed up your statement about the reliability but I have to disagree. I recently worked on a project that was using Selenium Chrome Driver and haven't encountered any reliability issues nor have I seen anything being dependant of screen resolution? You must be doing something wrong. Anyhow, since you want to use a different technique here's a quick info on how you would go about it:

Further I wonder if the latter is just a HTML response or if it's the the page's HTML that actual users would see

Since you're just starting you should equip yourself with a web debugging tool like Fiddler. It allows you to intercept and manipulate web requests and responses. There are plenty of free tutorials available on the web on how to use Fiddler so please find one and get up to speed with it - it shouldn't take long and Fiddler is the kind of tool that you can't go on without in a long run during web development or scraping.

The next step is to familiarize yourself with HttpClient (.NET 4.5+, or WebClient for earlier versions). It allows you to communicate with a website using fully customizable objects. You can tweak almost any aspect of a request or response: headers, cookies, protocols, etc. (so it all lines up with Fiddler). This class wraps a lot of logic already so you can focus on what's most important to you which is the communication.

I wonder how to send data via POST and how to get the entire HTML result of the web page.

Here's how you would use a HttpClient to download html string:

public Task<string> GetHtmlAsync(string url)
{
    try
    {
        var t = Task.Run(async () => {
            using (var client = new HttpClient())
            {
                var response = await client.GetAsync(url);
                return await response.Content.ReadAsStringAsync();
            }
        });
        return t;
    }
    catch (HttpRequestException e)
    {
        return Task.FromException<string>(e);
    }
}

It's pretty similar for POST but this has already been answered on StackOverflow: How to make HTTP POST web request