Call method that requires 'limit' and 'offset' in Linq

Question

Maybe I'm asking some kind of nonsense, but I don't understand how to build LINQ query if third-party method requires parameters limit and offset and returns total and items.

What I've come to so far:

Enumerable.
Range(0, maxPages /*1000*/).
Select(i => i * maxLimit /*96*/).
Select(offset => new {
    offset = offset,
    result = api3.SearchQuery(searchTerm, maxLimit, offset)
}).
TakeWhile(a => a.offset <= a.result.total).
SelectMany(a => a.result.items).
ToArray()

Obviously, this query makes one extra call of api3.SearchQuery (and result will be result.items.Length == 0, so theoretically a.offset <= a.result.total can be replaced with a.result.items.Length > 0).

Is there any way to get rid of extra call?

Answer 1

You're correct, theoretically a.offset <= a.result.total can be replaced with a.result.items.Length > 0, but there is one issue.

TakeWhile evaluates the predicate before yielding elements. This means it will make one extra fetch even if you know there are no more results.

One way to get the precise control you need is to define a custom TakeUntil method, like this one, which is heavily inspired by the one from MoreLinq.

public static class EnumerableExtensions
{
    public static IEnumerable<TSource> TakeUntil<TSource>(
        this IEnumerable<TSource> source,
        Func<TSource, bool> predicate)
    {
        ArgumentNullException.ThrowIfNull(source);
        ArgumentNullException.ThrowIfNull(predicate);

        return _(); IEnumerable<TSource> _()
        {
            foreach (var item in source)
            {
                yield return item;
                if (predicate(item))
                    yield break;
            }
        }
    }
}

You can simplify your method as follows:

Enumerable
    .Range(0, maxPages)
    .Select(i => i * maxLimit)
    .Select(offset => api3.SearchQuery(searchTerm, maxLimit, offset))
    .TakeUntil(a => a.Items.Count < maxLimit)
    .SelectMany(a => a.Items)
    .ToArray();