libcurl FTP-Download incomplete in the Unreal Engine

Question

I am currently trying to download a File from a public git-Repository using curl in my Unreal C++ Project. Here is the code I'm trying to execute that I derived from the FTP-Example:

// This is in the .h file
struct FFtpFile {
    FILE* File;
    const char* Filename;
};

void FtpFetch(const std::string URL, const char* Filename) {
    CURL* Curl = curl_easy_init();

    const FFtpFile FtpFile {
        nullptr,
        Filename
    };
    
    if (!Curl) {
        UE_LOG(LogTemp, Warning, TEXT("Error Initiating cURL"));
        return;
    }

    curl_easy_setopt(Curl, CURLOPT_URL, URL.c_str());
    curl_easy_setopt(Curl, CURLOPT_VERBOSE, 1);
    curl_easy_setopt(Curl, CURLOPT_FOLLOWLOCATION, 1);
    curl_easy_setopt(Curl, CURLOPT_SSL_VERIFYPEER, false);
    
    // Data Callback
    const auto WriteCallback = +[](void* Contents, const size_t Size, const size_t NumMem, FFtpFile* FileStruct) -> size_t {
        if (!FileStruct->File) {
            fopen_s(&FileStruct->File, FileStruct->Filename, "wb");
            if (!FileStruct->File) {
                return CURLE_WRITE_ERROR;
            }
        }
        return fwrite(Contents, Size, NumMem, FileStruct->File);
    };
    curl_easy_setopt(Curl, CURLOPT_WRITEFUNCTION, DownloadCallback);
    curl_easy_setopt(Curl, CURLOPT_WRITEDATA, FtpFile);

    
    const CURLcode Result = curl_easy_perform(Curl);
    if (Result != CURLE_OK) {
        const FString Message(curl_easy_strerror(Result));
        UE_LOG(LogTemp, Warning, TEXT("Error Getting Content of the Model File: %s"), *Message);
        return;
    }
    curl_easy_cleanup(Curl);

    // Close the Stream after Cleanup
    UE_LOG(LogTemp, Log, TEXT("Successfully Fetched FTP-File. Closing Write Stream"))
    if (FtpFile.File) {
        fclose(FtpFile.File);
    }
}

Note that this is executed on a separate Thread using the Unreal Async function:

void AsyncFetchModelFile(const std::string URL) {
    std::string Path = ...
    
    TFunction<void()> Task = [Path, URL]() {
        FtpFetch(URL, Path.c_str());
    };

    UE_LOG(LogTemp, Log, TEXT("Fetching FTP on Background Thread"))
    Async(EAsyncExecution::Thread, Task, [](){UE_LOG(LogTemp, Warning, TEXT("Finishied FTP on Background Thread!"))});
}

I already removed the curl_global calls, as the documentation states those are not thread-safe. I also tried running the code on the main thread, but the same error happens there too.

To the error itself: The download runs almost flawlessly, but the downloaded file (in this case a .fbx file) always misses the last ~800 Bytes and is therefore incomplete. Also, the file keeps being open in Unreal, so I can't delete/move the file unless I close the Editor.

Before writing this Unreal code I tried running the same code in a pure C++ setting and there it worked flawlessly. But for some reason doing the same in Unreal doesn't work.

I also tried using a private method instead of the lambda-Function, but that didn't make any difference.

Any help would be appreciated ~Okaghana

Answer 1

I didn't manage to get it working in the end (I suspect it's an Unreal-Related Bug), but I found another way using the included Unreal HTTP Module:

FString Path = ...

// Create the Callback when the HTTP-Request has finished
auto OnRequestComplete = [Path](FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful) {
    if (bWasSuccessful) {
        FFileHelper::SaveArrayToFile(Response->GetContent(), *Path);
        UE_LOG(LogTemp, Log, TEXT("Successfully downloaded the file to '%s'"), *Path)
    } else {
        UE_LOG(LogTemp, Warning, TEXT("Error downloading the file (See EHttpResponseCodes): %s"), Response->GetResponseCode())
    }
};


// Create a HTTP-Request and Fetch the file
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = FHttpModule::Get().CreateRequest();
Request->SetVerb("GET");
Request->SetURL(URL);
Request->OnProcessRequestComplete().BindLambda(OnRequestComplete);

Request->ProcessRequest();