How to build and stream a csv string as a file to a client (.net c#)

Question

I have the following method for talking a list of a SomeMetric model and converting it to a csv string. The method write the csv string as a file to HttpResponseMessage:

private HttpResponseMessage ConvertToCsvFileResponse(IEnumerable<SomeMetric> filterRecords, string fileName)
{
    var csvBuilder = new StringBuilder();

    //write the header
    csvBuilder.AppendLine("ColA,ColB,ColC,ColD,ColE");

    // Write the data lines.
    foreach (var record in filterRecords)
        csvBuilder.AppendFormat("{0},{1},{2},{3},{4}{5}", record.A, record.B, record.C, record.D, record.E, Environment.NewLine);

    // Convert to Http response message for outputting from api.
    HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
    result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
    result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment"); //attachment will force download
    result.Content.Headers.ContentDisposition.FileName = fileName;
    result.Content = new StringContent(csvBuilder.ToString());

    return result;
}

This method is called from my API controller and the response is output to the client as follows:

[HttpGet]
public HttpResponseMessage GetExampleCsv()
{
    try
    {
        var entries = _auditTableStorage.ListEntities<SomeMetric>();

        return ConvertToCsvFileResponse(entries.ToList(), "Example.csv");
    }
    catch (Exception ex)
    {
        return new HttpResponseMessage(HttpStatusCode.InternalServerError) {
            Content = new StringContent($"Problem occurred connecting to storage: {ex.Message}")
        };
    }
}

I generate the data by querying Azure Table Storage - but that's kind of irrelevant for my question.

The data comes from a data source in a paginated format. My current approach is to gather all the data together first (paging through until the end) to generate a list (IEnumerable is returned and .ToList(); is called). This data is then passed to the ConvertToCsv method.

As you can imagine, building the dataset up front in this way is Ok for small to medium size data sets BUT quickly becomes inefficient for large datasets.

Ideally, I'd like to modify my code to take chunks of the CSV string as it's being built and stream that down as file to the client making the request - just not sure how to convert what I have into a streamed version. Any points greatly appreciated!

NOTE: The code for generating csv here is only for demonstration purposes. I could integrate an object to CSV parser but wanted to show the string generation and dropping that out as it happens.

Answer 1

For anyone else looking to get an example of this, here's how I achieved streaming content to the client browser as it was being built:

private HttpResponseMessage ConvertToCsvFileResponse(IEnumerable<SomeMetric> filterRecords, string fileName)
{
    HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
    result.Content = new PushStreamContent((outputStream, httpContext, transportContext) =>
    {
        using (StreamWriter sWriter = new StreamWriter(outputStream, UnicodeEncoding.ASCII))
        {
            sWriter.Write("A,B,C,D,E");

            foreach (var record in filterRecords)
                sWriter.Write($"{Environment.NewLine}{record.A},{record.B},{record.C},{record.D},{record.E}");
        }

    });
    result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
    result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment"); 
    result.Content.Headers.ContentDisposition.FileName = fileName;

    return result;
}