C# get html from url. Error (429) unknown

Question

I am using the following code to retrieve the html code from a url. It is working fine for a url as:

• https://trends.google.com/trends/?geo=US However, not working for the following one, returning an error as

"The remote server returned an error: (429) unknown.'"

What could be the error or how to get more info about the error?

private void getHtmlFromUrl() {

    string urlAddress = "https://trends.google.es/trends/explore?q=test&geo=US";

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    if (response.StatusCode == HttpStatusCode.OK)
    {
        Stream receiveStream = response.GetResponseStream();
        StreamReader readStream = null;

        if (String.IsNullOrWhiteSpace(response.CharacterSet))
            readStream = new StreamReader(receiveStream);
        else
            readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

        string htmlData = readStream.ReadToEnd();

        response.Close();
        readStream.Close();
    }

}

Answer 1

It's no longer possible to get the source code of google services without an API because specifically the trends service makes many calls when you visit only trends.google.es/trends/explore?q=test hence the error 429.

Do not waste your time digging in proxies, browser emulation or bots none will work. The best way is to use Google API for c# .

Example Projects:

https://github.com/thegreymatter/GoogleTrends (Deprecated)

https://developers.google.com/api-client-library/dotnet/guide/aaa_oauth (Deprecated)

New Solution!

(We install python then convert our py script to an exe file to use from .net code. The good news is no api is needed with this method)

1- Install Python: https://www.python.org/downloads/windows/

2- Install Pytrends using:

pip install pytrends

3- Create a test.py and add support to arguments:

from pytrends.request import TrendReq
import sys

# Only need to run this once, the rest of requests will use the same session.
pytrend = TrendReq()
# Set your region
pytrend.geo = 'ES'
pytrend.build_payload(kw_list=[sys.argv[1]]) # Also supports an array of keywords e.g. kw_list=['test', 'music']
related_queries_dict = pytrend.related_queries()
print(related_queries_dict)
#Check the documentation for other available queries https://github.com/GeneralMills/pytrends#api-methods

Now if you cmd.exe (do not use PowerShell command won't work there) in the same location of test.py use the following command:

test.py test

Screenshot Results

4- Now let's convert test.py to an .exe (windows console file)

(a) Install Pyinstaller:

pip install pyinstaller

(b) Compile test.py to test.exe

pyinstaller test.py -F

5- Your "test.exe" is inside \dist\ directory. Let's see if it works:

Yes it does. (Keep in mind that the filesize of test.exe is 80 mb since it employs many libraries.)

Now all you have left to do is Process.Start the "test.exe" filename with argument "test" to read the output from your csharp app.

You can use IronPython instead to make python commands through your csharp code