Search code examples
vb.netwinformsproxywebclientwebrequest

WebClient or WebRequest to get the re-directed URL of the landing page


From the string I parsed from Bing's Pic of the Day, I got the info of the pic to be downloaded, let's say today it is /az/hprichbg/rb/PearlHarborWindows_EN-US8565186567, then we will have full URL of the image be like http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1366x768.jpg

Usually Bing has an image of higher resolutions, so I will download the image 1920x1200 too. It's easy with the URL changed to be like http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg, then give the task to a WebClient such as client1.DownloadFile(url, fileName)

The issue here is, some days, the resolution 1920x1200 is not available, and the download URL of this res.(1920x1200) will be re-directed to the URL of the image /sa/simg/hpb/NorthMale_EN-US8782628354_1920x1200.jpg - as default (you can check it).

So my try was a function to get the return/re-directed URL from the input URL:

 Public Function GetWebPageURL(ByVal url As String) As String
    Dim Request As WebRequest = WebRequest.Create(url)
    Request.Credentials = CredentialCache.DefaultCredentials
    Return Request.RequestUri.ToString 
 End Function

and compare to the input URL to see it they are different, but the result was not as expected.

Could anyone let me know the method to check this re-directed URL, like the return URL after we press Enter and wait for the site to load.

Please give me idea to overcome this obstacle. Thank you!

Notes: Some issues related to access rights on different PCs cause me not to use HttpWebRequest, so I prefer the solution not using HttpWebRequest (WebClient or others are better).


With help from @IvanValadares @AlenGenzić, and suggestion of Proxy for HttpWebRequest from @Jimi, I have come to the fair solution, as the below code:

    url1 = "http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg"

    Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(url1), HttpWebRequest)
    myHttpWebRequest.MaximumAutomaticRedirections = 1
    myHttpWebRequest.AllowAutoRedirect = True
    Dim defaultProxy As IWebProxy = WebRequest.DefaultWebProxy
    If (defaultProxy IsNot Nothing) Then
        defaultProxy.Credentials = CredentialCache.DefaultCredentials
        myHttpWebRequest.Proxy = defaultProxy
    End If

    Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse, HttpWebResponse)
    url2 = myHttpWebResponse.ResponseUri.ToString

    Label1.Text = url1
    Label2.Text = url2

Solution

  • With help from @IvanValadares @AlenGenzić, and suggestion of Proxy for HttpWebRequest from @Jimi, I have come to the fair solution as below:

        url1 = "http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg"
    
        Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(url1), HttpWebRequest)
        myHttpWebRequest.MaximumAutomaticRedirections = 1
        myHttpWebRequest.AllowAutoRedirect = True
        Dim defaultProxy As IWebProxy = WebRequest.DefaultWebProxy
        If (defaultProxy IsNot Nothing) Then
            defaultProxy.Credentials = CredentialCache.DefaultCredentials
            myHttpWebRequest.Proxy = defaultProxy
        End If
    
        Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse, HttpWebResponse)
        url2 = myHttpWebResponse.ResponseUri.ToString
    
        Label1.Text = url1
        Label2.Text = url2
    

    The System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required. is no longer thrown.