The underlying connection was closed; but works in browsers

Question

I have a simple web server running on a wifi chip (esp8266, written in LUA, running on nodeMCU). When i query the website in any browser, i get the response as expected and displayed in the browser (about 45 characters long). When i query it in C# or powershell, i get:

"The underlying connection was closed: The connection was closed unexpectedly."

I have tried numerous options suggested across many forums but none of them seem to have worked.

Is there any way possible to make a web request the same way IE or Chrome does? I'm not sure what extra steps browsers are doing internally such that they are able to get the response without issue? Why is this an issue in .NET?

My script is below. I am considering just using c# to fire off PhantomJS (headless browser), then using javascript to tell it to open the website, and then pass back the response. Or alternatively, use sockets to open a connection and do it that way, rather than relying on .NET wrappers.

# Set the useUnsafeHeaderParsing property to true
$netAssembly = [Reflection.Assembly]::GetAssembly([System.Net.Configuration.SettingsSection])
$bindingFlags = [Reflection.BindingFlags] "Static,GetProperty,NonPublic"
$settingsType = $netAssembly.GetType("System.Net.Configuration.SettingsSectionInternal")
$instance = $settingsType.InvokeMember("Section", $bindingFlags, $null, $null, @())
if($instance)
{
    $bindingFlags = "NonPublic","Instance"
    $useUnsafeHeaderParsingField = $settingsType.GetField("useUnsafeHeaderParsing", $bindingFlags)

    if($useUnsafeHeaderParsingField)
    {
        $useUnsafeHeaderParsingField.SetValue($instance, $true)
    }
}

# Try setting the certificate policy to a custom child class that always returns true
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@ 
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

# Try setting other attributes on the ServicePointManager class
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls -bxor [System.Net.SecurityProtocolType]::Ssl3
[System.Net.ServicePointManager]::Expect100Continue = $false;

# Initiate the web request
$r = [System.Net.WebRequest]::Create("http://192.168.1.7/GetStatusAsJson")
# Try long timeouts, with KeepAlive set to false; Also try giving it a user agent string etc.
$r.Timeout = 5000
$r.ReadWriteTimeout = 5000
$r.KeepAlive = $false
$r.Method = "GET"
$r.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36";
$r.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
$resp = $r.GetResponse()

Answer 1

I ended up using MSXML2.XMLHTTP.3.0 with vbscript, and executed the vbscript from powershell.

on error resume next
Dim oXMLHTTP
Dim oStream

Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP.3.0")
oXMLHTTP.Open "GET", WScript.Arguments.Item(2) & "/OpenDoor", False
oXMLHTTP.Send

If oXMLHTTP.Status = 200 Then
    Set oStream = CreateObject("ADODB.Stream")
    oStream.Open
    oStream.Type = 1
    oStream.Write oXMLHTTP.responseBody
    oStream.SaveToFile WScript.Arguments.Item(0) & "\OpenDoor.html"
    oStream.Close
End If