Search code examples
vb.netvisual-studiowebclientcookiecontainer

Vb.net webclient error : too many automatic redirections was attempted


This code was running just fine before , but since yesterday the error started to show up. After reading many similar disscutions i though that my problem was about cookies so i add a CookieContainer but i still get the same error. My code allows me to get pictures and their titles from a website and display them , here is my code:
Form 1 :

Private Sub Button5_Click(sender As Object, e As EventArgs) Handles Button5.Click
    Dim websiteURL1 As String = "http://www.gamestop.com/collection/upcoming-video-games"
    Class1.getPics(websiteURL1, "<img src=""(?<Data>[^>]*)""><p>(?<Dataa>[^>]*)<br>")
    Me.AutoScroll = True
End Sub

Class1 fucntion :

Public Shared Function getPics(website As String, pattern As String)
    Dim tempTitles As New List(Of String)()
    Dim tempTitles2 As New List(Of String)()
    Dim lestitres As New List(Of titlesclass)
    Dim webClient As New CookieAwareWebClient()
    webClient.Headers.Add("user-agent", "null")
    'If the website happens to go offline, at least your application wont crash.
    'Handle default proxy
    Dim proxy As IWebProxy = WebRequest.GetSystemWebProxy()
    proxy.Credentials = CredentialCache.DefaultCredentials
    webClient.Proxy = proxy

    Dim content As String = webClient.DownloadString(website)
    Dim query = From title In Regex.Matches(content, pattern).Cast(Of Match)
                Select New With {Key .Link = String.Concat("http://www.gamestop.com", title.Groups("Data").Value),
                     Key .Title = title.Groups("Dataa").Value}

    Dim titles = tempTitles.Distinct().ToArray() 'remove duplicate titles
    Dim titles2 = lestitres.Distinct().ToArray()
    lestitres.Clear()
    'Count Items
    Dim item As Integer = 0
    'Count Row
    Dim row As Integer = 0
    'image: 222*122
    For Each letitre In query.Distinct
        Dim ImageInBytes() As Byte = webClient.DownloadData(letitre.Link)
        Dim ImageStream As New IO.MemoryStream(ImageInBytes)
        Dim MyPic As New PictureBox
        Dim MyLab As New Label
        'x = numéro item fois largeur image
        'y = numéro de ligne fois hauteur image
        MyPic.Location = New Point(item * 222, row * 122)
        MyLab.Location = New Point(item * 222, row * 122)
        MyPic.SizeMode = PictureBoxSizeMode.AutoSize
        MyLab.Text = letitre.Title
        MyPic.Image = New System.Drawing.Bitmap(ImageStream)
        Form2.Controls.Add(MyPic)
        Form2.Controls.Add(MyLab)
        'Bring Labels to front
        For Each ctrl As Control In Form2.Controls
            'If the control is of type button
            If TypeOf ctrl Is Label Then
                'Then disable it
                ctrl.BringToFront()
            End If
        Next
        'Increment the item count
        item = item + 1
        'If item is multiple of 4 or could check 4 then
        If item Mod 4 = 0 Then
            'Reset counter
            item = 0
            'Increment Row
            row = row + 1
        End If
    Next
End Function

CookieContainer WebClient class :

    Imports System.Net
Imports System.Text.RegularExpressions
Public Class CookieAwareWebClient
    Inherits WebClient

    Private cc As New CookieContainer()
    Private lastPage As String

    Protected Overrides Function GetWebRequest(ByVal address As System.Uri) As System.Net.WebRequest
        Dim R = MyBase.GetWebRequest(address)
        If TypeOf R Is HttpWebRequest Then
            With DirectCast(R, HttpWebRequest)
                .CookieContainer = cc
                If Not lastPage Is Nothing Then
                    .Referer = lastPage
                End If
            End With
        End If
        lastPage = address.ToString()
        Return R
    End Function
End Class

Solution

  • The website did add a new function which detect automatically the webclient location (country) and redirect them to another website , so in my case i set my webclient proxy to the USA so it can visit the link i want to without redirections. To add the proxy settings you can use :

    webclient1.Proxy = New WebProxy("YourProxyServerName", port)
    

    For exemple

    webClient.Proxy = New WebProxy("100.12.34.36", 8080)
    

    Here is a list of free usa proxys : https://www.us-proxy.org/