Hi I'm sending a request through HttpClient to a webpage using this code:
Imports System.Net
Imports System.Net.Http
Public Class Form1
Dim client As HttpClient = New HttpClient()
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
' Set User-Agent header
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36")
' Send request and receive response
Dim response As HttpResponseMessage = client.GetAsync("https://ikalogs.ru/tools/map/?page=1&server=22&world=10&state=active&search=city&allies%5B1%5D=&allies%5B2%5D=&allies%5B3%5D=&allies%5B4%5D=&nick=Bauer&ally=&island=&city=&x=&y=").Result
If response.IsSuccessStatusCode Then
' Get response content as string
Dim vcmode As String = response.Content.ReadAsStringAsync().Result
' Print response content
RichTextBox1.Text = vcmode
End If
End Sub
End Class
I'm using the latest user agenti I'm taking from https://www.useragentstring.com/ but the reponse keep saying the browser is out of date:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>Browser is out of date!</title>
<meta charset="utf-8">
<meta name="description" content="Ikalogs - Saving battle reports" />
<meta name="author" content="ZigFreeD" />
<meta name="keywords" content="ikalogs, логовица, анализатор, ikariam" />
<meta name="copyright" content="(c) 2012-2015 by ZigFreeD"/>
<meta http-equiv="pragma" content="no-cache"/>
<meta name="language" content=""/>
<base target=”_blank” />
<noscript>
<meta http-equiv="refresh" content="0; url=/default/jsdisabled/">
</noscript>
<style>
*{
margin: 0;
padding: 0;
color: #542c0f;
font: 700 20px/27px Arial,sans-serif;
text-align: center;
text-decoration: none;
}
body{
background: url(/themes/default/img/themes/error/background.jpg) #eedbb2;
}
.errorBrowser{
width: 500px;
min-height: 190px;
position: fixed;
left: 50%;
top: 50%;
margin: -95px 0 0 -250px;
}
.errorIcoDesk{
font: italic 700 14px/18px Arial,sans-serif;
background: url(/themes/default/img/themes/browsers.png) no-repeat top left;
width: 50px;
float: left;
padding: 100px 25px 0;
cursor: pointer;
filter:progid:DXImageTransform.Microsoft.Alpha(opacity=80);
opacity: .8;
}
.errorIcoDesk:hover{
filter: progid:DXImageTransform.Microsoft.Alpha(opacity=100);
opacity: 1;
}
.errorFi{background-position: -100px 0;}
.errorCh{background-position: -200px 0;}
.errorOp{background-position: -300px 0;}
.errorEx{background-position: -400px 0;}
</style>
</head>
<body>
<div class="errorBG">
<div class="errorBrowser">
<h1>Your browser is outdated or does not support the necessary technology for the operation of the site.</h1>
<a href="//www.apple.com/safari/">
<div class="errorIcoDesk errorSa">
Apple Safari
</div>
</a>
<a href="//www.mozilla.com/firefox/">
<div class="errorIcoDesk errorFi">
Mozilla Firefox
</div>
</a>
<a href="//www.google.com/chrome/">
<div class="errorIcoDesk errorCh">
Google Chrome
</div>
</a>
<a href="//www.opera.com/">
<div class="errorIcoDesk errorOp">
Opera
</div>
</a>
<a href="//ie.microsoft.com/">
<div class="errorIcoDesk errorEx">
Internet Explorer
</div>
</a>
</div>
</div>
</body>
</html>
I've been trying differents user agents, but this is the latest I can find online and It seems quite weird that page doesn't accept this one. I think I am doing something wrong while adding the user agent to the header.
Quite important: never use
.Resultor.Wait()in this platform
You often need to configure your HttpClient a bit more.
Always better add a CookieContainer and specify to use Cookies (event though it's the default behavior when a CookieContainer exists).
Then also better add headers that specify that decompression is supported and what decompression methods are handled, otherwise you may get back garbage (not in this specific case, but, well, since you're there...)
Now the User-Agent can be almost anything that's recognized
Here, I'm using Lazy initialization, as Lazy<T>(Func<T>).
I find it useful, both because allows to in-line the configuration of the HttpClient object and also add a configured HttpClientHandler with a Lambda and because you may create the class that contains the HttpClient but never actually use it (the object is initialized when you request the Lazy<T>.Value)
Private Shared ReadOnly client As Lazy(Of HttpClient) = New Lazy(Of HttpClient)(
Function()
Dim client = New HttpClient(CreateHandler(True), True) With {.Timeout = TimeSpan.FromSeconds(60)}
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36")
client.DefaultRequestHeaders.Add("Cache-Control", "no-cache")
client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate")
Return client
End Function
)
Private Shared Function CreateHandler(autoRedirect As Boolean) As HttpClientHandler
Return New HttpClientHandler() With {
.AllowAutoRedirect = autoRedirect,
.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate,
.CookieContainer = New CookieContainer(),
.UseCookies = True ' Default here. To be clear...
}
End Function
After that, your HttpClient's request can act almost as a WebBrowser (unless the Web Site challenges your request and waits for a response - HSTS and different other tricks - then there's nothing you can do)
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim url = "https://..."
Dim response As HttpResponseMessage = Await client.Value.GetAsync(url)
If response.IsSuccessStatusCode Then
RichTextBox1.Text = Await response.Content.ReadAsStringAsync()
Else
Debug.WriteLine(response.StatusCode)
End If
End Sub
Protected Overrides Sub OnFormClosed(e As FormClosedEventArgs)
client.Value.Dispose()
MyBase.OnFormClosed(e)
End Sub
As a note, the Web Site you have in your post is not secure