.Net HttpWebRequest Response Returning Some crazy characters

Question

When I send a GET with this code:

Private Function SendGet(ByVal URL As String, ByRef CookieJar As CookieContainer, ByVal Method As String)
    Dim reader As StreamReader
    Dim responseString As String = ""
    Try

        Dim Request As HttpWebRequest = HttpWebRequest.Create(URL)
        Request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14"
        Request.CookieContainer = CookieJar
        Request.AllowAutoRedirect = False
        Request.Accept = "application/json, text/javascript, */*; q=0.01"
        Request.Headers.Add("Accept-encoding", "gzip, deflate")
        Request.Headers.Add("Accept-Language", "en-US,en;q=0.5")
        Request.CookieContainer = CookieJar
        Request.Host = "Removed"
        Request.Referer = "https://www.removed.com"
        Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"
        Request.Headers.Add("X-Requested-With", "XMLHttpRequest")
        Dim Response As HttpWebResponse = Request.GetResponse()

        For Each tempCookie In Response.Cookies
            CookieJar.Add(tempCookie)
        Next

        reader = New StreamReader(Response.GetResponseStream())
        responseString = reader.ReadToEnd()
        Response.Close()

    Catch ex As Exception
        MsgBox(ex.Message.ToString)
    End Try

The headers I'm sending are identical to what my browser sends as well. I am also (previously) picking up a cookie by logging in (different function) through POST, so the cookie container I'm sending does contain a valid cookie which i have also confirmed.

The response I'm seeing in my browser developer tools is nice clean JSON, but the code returns this craziness: (I've cut about 90% of it fyi). Is this something I'm going wrong or is there some way to decrypt this?

�      �X�o�8�W�<�S�KJ�з���=���mo���Lb��9:��~��B    W*�[�:�}����_gJ�%M�3���kx���3�u���O�?4'72�c:��3�L�4΅,�p�Le���9䆳ܹ���u4

Disclaimer: I have permission to automate this process

Answer 1

That stream looks like you're getting compressed data back from the server. That's because you are passing "accept-encoding" with gzip specified (eg - telling the server "send me compressed data if you support gzip").

So, either ditch your "accept-encoding" header, or implement a quick gzip decompression on the bytes returned. There is already another stack article that shows you how to do that in vb.net.