How to handle "The given header was not found" when paging records in c# API GET request?

Question

I'm requesting data from an API that requires paging records based on a custom header called "cursor". Only 100 records may be retrieved per call and as such I've created a while loop to execute. The loop functions... until it doesn't. Once all records are paged, the headers get dropped and my program errors out with:

"The given header was not found."

No inserts into my database occur because the requests are streamed until they are all paged.

How can I handle this so the program completes successfully?

The call being made is:

var products = await ProcessProducts(userAndPasswordToken, BaseUrl);

The task being called:

    private static async Task<List<Products>> ProcessProducts(string userAndPasswordToken, string BaseUrl)
    {
        //Construct urls
        string RequestPath = string.Format("food/products");
        string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        string CursorPath = string.Format("");

        //Use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
        var response = await client.GetAsync(FullUrl);

        //Extract string from the response right away
        var content = await response.Content.ReadAsStringAsync();

        //Pass it in instead of Steam.
        var Products = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Products>>(content);

        //Place the header key mapped with the cursor value.
        IEnumerable<string> values = response.Headers.GetValues("cursor");

        // string cursor = null;
        string cursor = "4146";
        // cursor = values?.FirstOrDefault();
        do
        {

            CursorPath = $"?cursor={cursor}";
            if (cursor == null) CursorPath = string.Format("");
            FullUrl = string.Format("{0}{1}{2}", BaseUrl, RequestPath, CursorPath);
            response = await client.GetAsync(FullUrl);
            content = await response.Content.ReadAsStringAsync();
            var nextProducts = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Products>>(content);

            // add Products in the next page.
            Products.AddRange(nextProducts);

            values = response.Headers.GetValues("cursor");
            cursor = values?.FirstOrDefault();

            System.Threading.Thread.Sleep(1000);
            Console.WriteLine(FullUrl);
            Console.WriteLine(response);

        } while (cursor != null);

        return Products;
    }

The insert happens here with a SQL stored procedure:

            using (SqlConnection conn = new SqlConnection(lConnString))
            {
                conn.Open();
                using (TransactionScope ts = new TransactionScope())
                {

                    foreach (var repo in products)
                    {
                        SqlCommand cmdIns = new SqlCommand("usp_insert_Products", conn);
                        cmdIns.CommandType = CommandType.StoredProcedure;
                        cmdIns.Parameters.AddWithValue("@ProductId", repo.ProductId.ToString() ?? (object)DBNull.Value);
                        cmdIns.Parameters.AddWithValue("@ProductDetailId", repo.ProductDetailId.ToString() ?? (object)DBNull.Value);
                        cmdIns.ExecuteNonQuery();
                    }

                    ts.Complete();
                }
                conn.Close();

The sample code I have been provided has a completely different method, which I am on the cusp of moving to if I can't get my code to function properly:

Sample code

// Paging is handled by making a request and then making
// follow up requests as long as a "cursor" is returned.
string cursor = null;
do
{
    var query = HttpUtility.ParseQueryString("");
    query["locationId"] = "1";
    query["businessDate"] = "2019-04-30";
    if (cursor != null) query["cursor"] = cursor;

    var fullUrl = $"{url}/{endpoint}?{query}";
    _testOutputHelper.WriteLine(fullUrl);
    var json = client.DownloadString(fullUrl);
    results.AddRange(
        JsonConvert.DeserializeObject<List<Check>>(json));

    cursor = client.ResponseHeaders["cursor"];
} while (cursor != null);
}

Answer 1

You're using response.Headers.GetValues("cursor") which is documented to throw an InvalidOperationException if the header can't be found.

You should use TryGetValues instead:

if (response.Headers.TryGetValues("cursor", out IEnumerable<string> values))
{
    // header exists
}
else
{
    // header doesn't exist
}