We have an Angular App that allow user to download a list of image through their ids in a database. The ASP.NET Core Web API uses the response stream to create the image zip file and transfer it while appending image to the archive.
The problem is that for large archive files (my test is 2.5 Gb with around 2500 images), the response body (client side) is always null. No problem for smaller archive files.
Some relevant code:
First, on the client side, this is the trigger in the component:
modalInstance.componentInstance.onConfirmEvent.subscribe(res => {
this.downloadInProgress = true;
// workingList contains a list of image ids
this.prepareDownloadByObject(workingList, res.createFileTree, res.createMetadata)
})
In the download service, we use HttpClient to post an object which has a list of image ids, then check the download progress with handleDownloadProgress. Once the event type is Response, we use the body to create an object url and download the file (this method is everywhere in SO):
prepareDownloadByObject(workingList, createFileTree, createMetadata) {
this.downloadByWorkingList(workingList, createFileTree, createMetadata).subscribe(
res => this.handleDownloadProgress(res, workingList),
async err => this.handleDownloadError(err)
)
}
downloadByWorkingList(workingListDto: IWorkingList,createTreeDirectory:boolean, createMetadata: boolean): Observable<any> {
return this.http.post(`${this.baseUrl}WorkingList/DownloadWorkingList/${createTreeDirectory}/${createMetadata}`,workingListDto,
{
observe: 'events',
responseType: "arraybuffer",
reportProgress: true
});
}
handleDownloadProgress(event, workingList?: IWorkingList) {
if (event.type === HttpEventType.DownloadProgress) {
//[... UI update ...]
} else if (event.type === HttpEventType.Response) {
this.downloadFile(event, workingList);
//[... UI update ...]
} else {
console.log("Unknown event type", event);
}
downloadFile(data: any, workingList?: IWorkingList) {
// true
if (data.body == null) {
return;
}
const downloadedFile = new Blob([data.body], { type: data.body.type });
const a = document.createElement("a");
a.setAttribute("style", "display:none;");
document.body.appendChild(a);
a.download = "workingList" +"_"+ this.getFileDateFormat() + ".zip";
a.href = URL.createObjectURL(downloadedFile);
a.target = "_blank";
a.click();
document.body.removeChild(a);
}
API side - controller:
[HttpPost("{createTreeDirectory}/{createMetadata}")]
public async Task<IActionResult> DownloadWorkingList(WorkingListDto workingListDto, bool createTreeDirectory, bool createMetadata, CancellationToken token)
{
try
{
if (workingListDto.ImageIds == null || workingListDto.ImageIds.Count == 0)
{
return BadRequest("You have to select images to download");
}
Response.ContentType = "application/octet-stream";
Response.Headers.Append("Content-Disposition", "attachment; filename=workingList.zip");
await Response.StartAsync(token);
await this._workingListAppService.DownloadWorkingList(workingListDto, createTreeDirectory, createMetadata, token, Response.BodyWriter.AsStream()).ConfigureAwait(false);
await Response.CompleteAsync();
return new EmptyResult();
}
catch (ArgumentException ex)
{
return this.BadRequest(new { ex.Message, ex.StackTrace });
}
catch (Exception ex)
{
return this.Problem(ex.Message, "WorkingList/DownloadAndSaveWorkingList", 500);
}
}
WorkingListAppService gathers the images filepath (on an Azure fileshare) and transfers it to a specific service. This service will get the images byte content, write it as an entry into a zipArchive which is build directly into the response stream. This method is used to avoid writing the file on disk or load it fully in memory:
public async Task<Stream> CreateImageZipFileStream(List<Image> image, string user, CancellationToken token, Stream stream, bool createTreeDirectory = true, bool createMetadataFile = false)
{
// In this case, stream is Response.BodyWriter.AsStream()
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, true))
{
foreach (var image in images)
{
await AddImageToArchive(archive, createTreeDirectory, createMetadataFile, image, token);
}
}
// Irrelevant here
return stream;
}
private async Task AddImageToArchive(ZipArchive archive, bool createTreeDirectory, bool createMetadata, Image imageItem, CancellationToken token)
{
string entryName = imageItem.UserFriendlyName + Constantes.IMAGE_EXTENSION;
if (createTreeDirectory)
{
// Change the entryName to add a subfolder in the archive
[...]
}
var entry = archive.CreateEntry(entryName, CompressionLevel.NoCompression);
using (var entryStream = entry.Open())
{
try
{
// This methods get the file bytes from an azure fileshare and write it to entryStream
await this._imageFileShareClient.GetFileContent(imageMaterialFolder + "/" + imageItem.ApplicationCamera.SiteRef.SiteName, imageItem.FilePath, entryStream);
}
catch
{
try
{
entryStream.Position = 0;
// This methods get a fallback file bytes from an azure fileshare and write it to entryStream
await this._imageFileShareClient.GetFileContent(fisheyedImageFolder + "/" + imageItem.ApplicationCamera.SiteRef.SiteName, imageItem.FilePath, entryStream);
}
catch
{
this._logger.LogError("An image requested for download was missing from fileshare. ImageId: {0}", imageItem.Id);
// Do not throw on error, Ignore missing image and continue
}
}
}
}
Some more information: the request is completely done, data are transmitted to client: In the network tab of the console the request size is the size of the file.
We tried arraybuffer, blob request responseType with the same result.
We tried application/zip, application/octet-stream as sesponse Content-Type.
The library file saver got the same problem, event body being null.
Reminder of the problem
When the API reaches Response.CompleteAsync(), the client gets the event typed Response. So the Angular app tries to access the body of the response which is null. This is the main question.
Another subsidiary question which could work around the problem: is there a way to avoid loading the entire file in memory client side?
Thanks to @browsermator indications I managed to make the navigator handle the download.
The main problem was authentication, which I bypassed with a token to allow the user on next download.
For the POST, I generate a form like this:
<form ngNoForm action="{{baseUrl + 'WorkingList/DownloadWorkingList/' +createFileTree + '/'+ createMetadata + '/' + token}}" method="post">
<input *ngFor="let imageId of workingList.imageIds; let index = index" type="hidden" name="imageIds[]" [(ngModel)]="workingList.imageIds[index]"/>
<button type="submit" class="btn btn-primary" rippleEffect>
{{ "DOWNLOAD_MODAL.CONFIRM" | translate }}
</button>
</form>
For the GET (not part of the question), I generate a URL the same way as the Form submit url, just adding the workingList Id as parameter.
Server side I added a check on the token added in the query url, and added [FromForm] on the object parameter.
This allow the navigator to handle the response as a file download and let him handle resources.
This is probably not the best (authentication is by passed, and user auth has to be handled manually) but it solved my problem and we can deal with this authorization method.
PS: I did not use the File() method in the controller because it close the stream too soon for the archive to be build