I am using .net 4.61 and doing a httpwebrequest with VB on https://www.tvone.tv and it fails to connect to server every time. Any other lookup on any other domain other than .tv works fine. I got http://ustream.tv to work once but only once and never again.
Is this a .net problem or a dns problem with .net?
IE, Edge, Firefox and Chrome all load the webpage fine from the computer I use to compile on. So, I know it isn't a network problem.
Is there a specific way I need to load .tv domains with httpwebrequests?
One of the sites you mentioned had a Https: schema in the URI.
This implies that a security protocol is used. Upon inspection, this site uses TLS 1.2 security protocol. So you may need to enable it (required in Windows 7 when targeting a .Net Framework version prior to 4.8, usually not required in Windows 10, since it's the default), setting the ServicePointManager SecurityProtocol:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Also, most of the time, sites use Cookies (or have to use them, because external resources require them to. Google tools, for example).
You have to provide means to accept, store and exchange them.
This is accomplished instantiating a CookieContainer and passing it to the HttpWebRequest using its CookieContainer Property Property.
So I tested all of them to see what was happening behind the curtains (see the results).
One of them is more interesting than the others, (https://www.tvone.tv). It's one of those that don't accept a schema other than Https:. If you change it, it will redirect your request back to it (forcing security with Tls 1.2 certificate exchange). But it doesn't (the home page, at least) use cookies.
Since I think it's somewhat interesting, I'm posting the code used for testing. If, in the Headers setup, you set .AllowAutoRedirect = False, you can follow the redirection and see what it does.
Host URI: https://tvone.tv/ - IP Address: 45.79.161.20
Http Protocol: Http/1.1
Response Status: OK
Cookies n°: 0
Security Protocol: Tls 1.2
Try this with Https: schema
Host URI: http://www.ustream.tv/ - IP Address: 199.66.238.212
Http Protocol: Http/1.1
Response Status: OK
Cookies n°: 2
Security Protocol: N/A
Host URI: http://www.fyi.tv/ - IP Address: 151.101.14.168
Http Protocol: Http/1.1
Response Status: OK
Cookies n°: 1
Security Protocol: N/A
This is the code used for testing.
(It's a redacted/translated redux of a library of mine, so you don't have to worry about stepping on anyone's toes if you use it).
The main function, HTTP_GetWebPageAsync(), can be called like this:
(The object passed is returned with its properties filled with the data shown in the results above, plus the whole Html page (in Public Property PayLoad) of the response).
Dim httpReqObject As HttpReqObj = New httpReqObject()
Dim maxWaitTimeOut As Integer = 30 'Seconds
httpReqObject.SiteURL = "https://tvone.tv/"
httpReqObject = Await HTTP_GetWebPageAsync(httpReqObject, maxWaitTimeOut)
Imports System.IO
Imports System.Net
Imports System.Net.Security
Imports System.Reflection
Imports System.Security
Imports System.Security.Authentication
Imports System.Security.Cryptography.X509Certificates
Imports System.Text
Private Const COR_E_INVALIDOPERATION As Int32 = &H80131509
Public Class HttpReqObj
Public Property SiteURL As String
Public Property StatusCode As HttpStatusCode
Public Property PayLoad As String
Public Property Cookies As CookieContainer
Public Property HostAddress As IPAddress()
Public Property HttpProtocol As String
Public Property SslProtocol As SslProtocols
End Class
Public Async Function HTTP_GetWebPageAsync(HttpSite As HttpReqObj, ReqTimeOut As Integer) As Task(Of HttpReqObj)
Dim httpRequest As HttpWebRequest
Dim cookiejar As New CookieContainer()
Dim statusCode As HttpStatusCode
Dim redirectURL As String = String.Empty
Dim referer As String = String.Empty
Dim payload As String = String.Empty
Dim maxAwaitableTimeout As Integer = ReqTimeOut
Dim maxHops As Integer = 40
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
ServicePointManager.ServerCertificateValidationCallback = AddressOf TlsValidationCallback
Try
httpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
HttpSite.HostAddress = Await Dns.GetHostAddressesAsync(httpRequest.Host)
HTTP_RequestHeadersInit(httpRequest, cookiejar, "")
httpRequest.Method = WebRequestMethods.Http.Get
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
statusCode = httpResponse.StatusCode
payload = If(statusCode = HttpStatusCode.OK, ProcessResponse(httpResponse, HttpSite), String.Empty)
redirectURL = URIFromResponseLocation(httpResponse).ToString()
HttpSite.HttpProtocol = "Http/" + httpRequest.ProtocolVersion.ToString()
End Using
'On Page Redirection, follow redirections [maxHops] hops and maxAwaitableTimeout] time
If (redirectURL <> HttpSite.SiteURL) Or
(statusCode = HttpStatusCode.Moved) OrElse
(statusCode = HttpStatusCode.Found) OrElse
(statusCode = HttpStatusCode.RedirectMethod) Then
If (redirectURL <> HttpSite.SiteURL) Then
Dim sw As Stopwatch = New Stopwatch()
sw.Start()
referer = HttpSite.SiteURL
HttpSite.SiteURL = redirectURL
httpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
httpRequest.Method = WebRequestMethods.Http.Get
HTTP_RequestHeadersInit(httpRequest, cookiejar, referer)
Dim hops As Integer = 1
Do
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
payload = ProcessResponse(httpResponse, HttpSite)
statusCode = httpResponse.StatusCode
redirectURL = URIFromResponseLocation(httpResponse).ToString()
End Using
If redirectURL <> HttpSite.SiteURL Then
If (statusCode = HttpStatusCode.Moved) OrElse
(statusCode = HttpStatusCode.Found) OrElse
(statusCode = HttpStatusCode.RedirectMethod) Then
HttpSite.SiteURL = redirectURL
httpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
HTTP_RequestHeadersInit(httpRequest, cookiejar, referer)
hops += 1
End If
End If
If sw.Elapsed.Seconds > maxAwaitableTimeout Then
statusCode = HttpStatusCode.RequestTimeout
Exit Do
End If
Loop While (statusCode <> HttpStatusCode.OK) AndAlso (hops < maxHops)
sw.Stop()
End If
End If
Catch exW As WebException
statusCode = If(exW.Response IsNot Nothing,
CType(exW.Response, HttpWebResponse).StatusCode,
CType(exW.Status, HttpStatusCode))
payload = String.Empty
Catch exS As System.Exception
If exS.HResult = COR_E_INVALIDOPERATION Then
statusCode = CType(WebExceptionStatus.ConnectFailure, HttpStatusCode)
Else
statusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)
End If
payload = String.Empty
Finally
ServicePointManager.ServerCertificateValidationCallback = Nothing
End Try
HttpSite.Cookies = cookiejar
HttpSite.PayLoad = payload
HttpSite.StatusCode = statusCode
Return HttpSite
End Function
Private Function TlsValidationCallback(sender As Object, CACert As X509Certificate, CAChain As X509Chain, ssl_PolicyErrors As SslPolicyErrors) As Boolean
If ssl_PolicyErrors = SslPolicyErrors.None Then
Return True
End If
Dim _Certificate As New X509Certificate2(CACert)
'Dim _CACert As New X509Certificate2("./ca.cert") 'Add your certificate here
'CAChain.ChainPolicy.ExtraStore.Add(_CACert)
CAChain.Build(_Certificate)
For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
(CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
Return False
End If
Next
Return True
End Function
Private Function ExtractSslProtocol(stream As Stream) As SslProtocols
Dim flags As BindingFlags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic Or BindingFlags.Static
Dim objConnection = stream.[GetType]().GetField("m_Connection", flags).GetValue(stream)
Dim objTlsStream = objConnection.[GetType]().GetProperty("NetworkStream", flags).GetValue(objConnection)
Dim objSslState = objTlsStream.[GetType]().GetField("m_Worker", flags).GetValue(objTlsStream)
Return CType(objSslState.[GetType]().GetProperty("SslProtocol", flags).GetValue(objSslState), SslProtocols)
End Function
Private Sub HTTP_RequestHeadersInit(ByRef HttpReq As HttpWebRequest, CookieJar As CookieContainer, Referer As String)
HttpReq.ServicePoint.MaxIdleTime = 30000
HttpReq.ServicePoint.Expect100Continue = False
HttpReq.Timeout = 30000
HttpReq.CookieContainer = CookieJar
HttpReq.KeepAlive = True
HttpReq.AllowAutoRedirect = False
HttpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
HttpReq.Referer = Referer
HttpReq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
HttpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
HttpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.8,en;q=0.5")
HttpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
HttpReq.Headers.Add(HttpRequestHeader.CacheControl, "max-age=0")
End Sub
Private Function ProcessResponse(Response As HttpWebResponse, HttpSite As HttpReqObj) As String
Dim html As String
Dim enc As Encoding = Encoding.UTF8
Try
If Response.CharacterSet.Length > 0 Then
enc = Encoding.GetEncoding(Response.CharacterSet)
End If
Using rstream As Stream = Response.GetResponseStream()
HttpSite.SslProtocol = ExtractSslProtocol(rstream)
Using ms As New MemoryStream()
If Response.ContentEncoding.ToLower().Contains("gzip") Then
Using gzip As New Compression.GZipStream(rstream, Compression.CompressionMode.Decompress)
gzip.CopyTo(ms)
End Using
ElseIf Response.ContentEncoding.ToLower().Contains("deflate") Then
Using deflate As New Compression.DeflateStream(rstream, Compression.CompressionMode.Decompress)
deflate.CopyTo(ms)
End Using
Else
rstream.CopyTo(ms)
End If
ms.Position = 0
Using reader As New StreamReader(ms, enc)
html = reader.ReadToEnd().Trim()
End Using
End Using
End Using
Catch _Ex As Exception
Return String.Empty
End Try
Return html
End Function
Private Function URIFromResponseLocation(RefResponse As HttpWebResponse) As System.Uri
Dim url As Uri
Dim loc As String = RefResponse.Headers("Location")
Try
If Uri.IsWellFormedUriString(loc, UriKind.Absolute) Then
url = New Uri(loc, UriKind.Absolute)
Else
Dim hostUri As String = RefResponse.ResponseUri.GetComponents(
UriComponents.SchemeAndServer, UriFormat.Unescaped) + loc
If Uri.IsWellFormedUriString(hostUri, UriKind.Absolute) Then
url = New Uri(hostUri)
Else
url = New Uri(RefResponse.ResponseUri.GetComponents(
UriComponents.Scheme, UriFormat.Unescaped) +
RefResponse.ResponseUri.Host + loc
)
End If
End If
Catch _Ex As Exception
url = New Uri(loc, UriKind.Relative)
End Try
Return url
End Function