Dynamically allocate memory for storage of HTML source downloaded with WinHttpReadData

Question

First of all, string cannot be used, this is a requirement.

I am attempting to implement Winhttp in order to download content from HTTP. I have used the example provided on MSDN (http://msdn.microsoft.com/en-us/library/windows/desktop/aa384270(v=vs.85).aspx).

As im sure some of you are aware WinHttpReadData() reads data into a temp buffer, over writing existing data until the request is completed. This is fine if you simply want to print out the buffer each time, however, i need to store the entire response into a buffer for use later.

For this i have created a struct which contains all the "things" i need to perform the request, i pass this struct by reference to the function which performs the request. The struct is as follows:

struct HttpG
{
    wchar_t*    wszUserAgent; 
    wchar_t*    wszCookie;
    wchar_t*    wszHost;
    wchar_t*    wszPath; 
    char*       szResponse;
};

The function that does the request is defined as the following:

int HttpGet(HttpG &http_get);

So far so good....

Now the problem arises when i try to allocate memory dynamically for http_get.szResponse. Not all data is getting read. I'm not going to post the entire example code from MSDN, but i will post the part of the code that is giving me the issue. If you look at the MSDN link above you will see which part of the code i am referring to. This is the main loop that downloads the data.

// Read the Data.
ZeroMemory(szOutBuffer, dwSize + 1);

if(!WinHttpReadData(hRequest, (LPVOID)szOutBuffer, dwSize, &dwDownloaded))
{                                  
    OutputDebugStr("Error in WinHttpReadData\n");   
}
else
{
     // Read data here              
     if(http_get.szResponse == NULL)
     {                  
         // This part seems to work as needed
         http_get.szResponse = new char[dwSize + 1];        
         ZeroMemory(http_get.szResponse, dwSize + 1);
         strcpy(http_get.szResponse, szOutBuffer);
         http_get.szResponse[dwSize + 1] = '\0';            
     }              
     else
     {
         // Im sure the problems is here, full source
         // is not getting put into http_get.szResponse.

         // Create temp buffer
         szTemp = new char[strlen(http_get.szResponse) + 1];    
         ZeroMemory(szTemp, strlen(http_get.szResponse) + 1);
         strcat(szTemp, http_get.szResponse);                   

         // Resize origonal buffer to hold new data                 
         http_get.szResponse = new char[strlen(szTemp) + dwSize + 1];
         ZeroMemory(http_get.szResponse, strlen(szTemp) + dwSize + 1);
         strcpy(http_get.szResponse, szTemp);
         strcat(http_get.szResponse, szOutBuffer);
         http_get.szResponse[strlen(szTemp) + dwSize + 1] = '\0';               
     }              
}           

// Free the memory allocated to the buffer.
delete[] szTemp;
delete[] szOutBuffer;           

// This condition should never be reached since WinHttpQueryDataAvailable
// reported that there are bits to read.
if(!dwDownloaded)
{
    break;
}

I create the struct and call function like this:

HttpG http_get;
http_get.wszHost = L"au.yahoo.com";
http_get.wszPath = L"/?p=us";
http_get.wszUserAgent = L"Blah blah blah";
http_get.szResponse = NULL;

HttpGet(http_get);  

So essentially at the end of the request i want all the data to be inside http_get.szResponse. Sorry if this is a bit messy/vague, i tried to explain it as best as possible. What am i doing wrong? Have been stuck on this all day, any help is highly appreciated.

Thanks guys.

Answer 1

You must call WinHttpReadData() in a loop until there is no more data to read, and you need to dynamically (re)allocate your response buffer on each iteration of that loop. If the requirements forbid you from using std::string then they likely forbid you from using std::vector as well, so you will have to resort to manual memory management, eg:

struct HttpG
{
    wchar_t*    wszUserAgent; 
    wchar_t*    wszCookie;
    wchar_t*    wszHost;
    wchar_t*    wszPath; 
    u_char*     ucResponse;
    int         ucResponseSize;
};

u_char ucBuffer[1024], *ucTemp;    
DWORD dwDownloaded;

do
{
    if (!WinHttpReadData(hRequest, ucBuffer, sizeof(ucBuffer), &dwDownloaded))
    {                                  
        OutputDebugStr("Error in WinHttpReadData\n");   
        break;
    }

    if (dwDownloaded == 0)
        break;

    if (http_get.ucResponse == NULL)
    {                  
        http_get.ucResponse = new u_char[dwDownloaded];        
        memcpy(http_get.ucResponse, ucBuffer, dwDownloaded);
        http_get.ucResponseSize = dwDownloaded;            
    }              
    else
    {
        ucTemp = new u_char[http_get.ucResponseSize + dwDownloaded];    
        memcpy(ucTemp, http_get.ucResponse, http_get.ucResponseSize);                   
        memcpy(&ucTemp[http_get.ucResponseSize], ucBuffer, dwDownloaded);                   

        delete[] http_get.ucResponse;
        http_get.ucResponse = ucTemp;               
        http_get.ucResponseSize += dwDownloaded;
    }              
}
while (true);           

HttpG http_get;
http_get.wszHost = L"au.yahoo.com";
http_get.wszPath = L"/?p=us";
http_get.wszUserAgent = L"Blah blah blah";
http_get.ucResponse = NULL;
http_get.ucResponseSize = 0;

HttpGet(http_get);

// use ucResponse up to ucResponseSize bytes as needed...

delete[] http_get.ucResponse;