I have a function that gets x numbers og items from a SharePoint list. It takes the items in batches. After each batch I do something with the items, destroys everything and take the next batch for calculation. I currently consider to use events. So have an event for each batch. Is this the correct strategy or is there a better way to do it? I am thinking anonymous functions or something similar?
public static List<Item> GetAllItems(this List list, int rowLimit, List<string> fields, bool includeRoleAssignments, ILogger logger)
{
var result = new List<Item>();
var ctx = list.Context;
ListItemCollectionPosition position = null;
var camlQuery = new CamlQuery();
camlQuery.ViewXml =
@"<View Scope='RecursiveAll'>
<Query>
<OrderBy Override='TRUE'><FieldRef Name='ID'/></OrderBy>
</Query>
<ViewFields></ViewFields>" +
"<RowLimit Paged='TRUE'>" + rowLimit + "</RowLimit>" +
"</View>";
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
do
{
try
{
using (var clonedCtx = ctx.Clone(ctx.Url))
{
List listWithClonedContext = clonedCtx.Web.Lists.GetByTitle(list.Title);
clonedCtx.Load(listWithClonedContext);
clonedCtx.ExecuteQuery();
ListItemCollection listItems = null;
camlQuery.ListItemCollectionPosition = position;
listItems = listWithClonedContext.GetItems(camlQuery);
foreach (string field in fields)
{
clonedCtx.Load(listItems, includes => includes.Include(i => i[field]));
}
if (!includeRoleAssignments) {
clonedCtx.Load(listItems, item => item.ListItemCollectionPosition);
}
else {
clonedCtx.Load(listItems, item =>
item.ListItemCollectionPosition,
item => item.Include(
i => i.RoleAssignments.Include(
ra => ra.Member,
ra => ra.Member.LoginName,
ra => ra.RoleDefinitionBindings.Include(rd => rd.Description, rd => rd.Name))));
}
clonedCtx.Load(listItems, item => item.ListItemCollectionPosition);
clonedCtx.ExecuteQueryWithIncrementalRetry(3, 1, logger);
// here i want to do something with items before next loop/batch
position = listItems.ListItemCollectionPosition;
if (position != null)
{
logger.WriteTrace(string.Format("Iteration on getting items performed: {0}", position.PagingInfo), SeverityLevel.Verbose);
}
else
{
logger.WriteTrace("Getting all items finished.", SeverityLevel.Verbose);
}
logger.Flush();
}
}
catch (Exception ex)
{
logger.WriteException(ex);
}
}
while (position != null);
return result;
}
Maybe events are an option but there might also be an easier way to "stream" them out instead of returning all at once with a list. Therefore use yield
and change to IEnumerable<Item>
:
public static IEnumerable<Item> EnumerateItems(this List list, int rowLimit, List<string> fields, bool includeRoleAssignments, ILogger logger)
{
// ...
do
{
try
{
using (var clonedCtx = ctx.Clone(ctx.Url))
{
//...
camlQuery.ListItemCollectionPosition = position;
listItems = listWithClonedContext.GetItems(camlQuery);
// ...
foreach(Item x in listItems)
{
yield return x;
}
position = listItems.ListItemCollectionPosition;
// ...
}
while (position != null);
}
On this way you can start processing them while you are still fetching them or you could filter them, for example with Where
,Skip
or Take
without loading all into memory first.