.net Can not download image. Request gets blocked, why? (httpwebrequest)

Question

On some domains I can not download images. The code is working with other domains, but with some its doesn't. I can't figure out why. I can download the image with a full browser like firefox or chrome but not with httpwebrequest. I tried to emulate a full browser as close as possible but no success.

Maybe you can help me to find the error?

Error is:

A System.Net.WebException exception error has occurred in System.dll. Additional Information: The underlying connection has been closed: Unexpected error while sending

On line:

    Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)

Here is the code with sample of failling download:

    Dim httpWebRequest = DirectCast(WebRequest.Create("https://www.oglf.org/wp-content/uploads/2017/12/BestLEDTeethWhitening.jpg"), HttpWebRequest)
    httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0"
    httpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    httpWebRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;q=0.5")
    httpWebRequest.Referer = "https://www.oglf.org"
    httpWebRequest.AllowAutoRedirect = True
    httpWebRequest.KeepAlive = True

    Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)

    If (httpWebResponse.StatusCode <> HttpStatusCode.OK AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Moved AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Redirect) OrElse Not httpWebResponse.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase) Then
        Return
    End If
    Using stream = httpWebResponse.GetResponseStream()
        Using fileStream = File.OpenWrite("c:\imagetest.jpg")
            Dim bytes = New Byte(4095) {}
            Dim read = 0
            Do
                If stream Is Nothing Then
                    Continue Do
                End If
                read = stream.Read(bytes, 0, bytes.Length)
                fileStream.Write(bytes, 0, read)
            Loop While read <> 0
        End Using
    End Using

Answer 1

In this case, the problem is that the ssl connection negotiation is failing. If is it of no consequence to you that you don't use ssl to connect, I would just use the following:

Private Sub DownloadImage(ByVal source As String, destination As String)
    If source.StartsWith("https://") Then source = source.Replace("https://", "http://")
    Using wc As New System.Net.WebClient
        wc.DownloadFile(source, destination)
    End Using
End Sub

If you must use ssl, try setting the TLS version to 1.2 since some websites require that, and the .net web client doesn't default to that:

Private Sub DownloadImage(ByVal source As String, destination As String)
    System.Net.ServicePointManager.SecurityProtocol = Net.SecurityProtocolType.Tls12
    Using wc As New System.Net.WebClient
        wc.DownloadFile(source, destination)
    End Using
End Sub

Called using:

DownloadImage("https://www.oglf.org/wp-content/uploads/2017/12/BestLEDTeethWhitening.jpg", _
              "c:/tmp/test.jpg")