Data migration for a huge amount of pages

Question

I have a base page with the Field1 and SomeBlock

public abstract class BasePage : PageData
{
    [CultureSpecific]
    [Display(Name = "Field1")]
    public virtual string Field1 { get; set; }


    [Display(Name = "SomeBlock")]
    public virtual SomeBlock SomeBlock { get; set; }
}
public class SomeBlock : BlockData
{
    [CultureSpecific]
    [Display(Name = "Field1")]
    public virtual string Field1 { get; set; }
}

Now I have to move the Field1 into SomeBlock inside the BasePage and move all existing data from BasePage.Field1 to BasePage.SomeBlock.Field1

For this purposes I have created a job that looks like:

    private void MigrateFields()
    {
        var repo = ServiceLocator.Current.GetInstance<IContentRepository>();

        var descendents  = repo.GetDescendents(_root).Where(p => SafeGetContent(repo, p) is BasePage).Select(repo.Get<BasePage>);

        foreach (var basePage in descendents)
        {

            BasePage writeablePage = (BasePage)basePage .CreateWritableClone();

            if (basePage.SomeBlock == null)
                basePage.SomeBlock = new SomeBlock ();

            if (string.IsNullOrWhiteSpace(basePage.SomeBlock.Field1))
            {
                basePage.SomeBlock.Field1 = basePage.Field1;
            }


            DataFactory.Instance.Save(writeablePage, SaveAction.Publish, AccessLevel.NoAccess);
        }
    }

Everything works just fine if I have about 1000 of pages. However if the site has >20000 it looks like the job just dies.

Answer 1

There is an issue with

var descendents  = repo.GetDescendents(_root)
.Where(p => SafeGetContent(repo, p) is BasePage).Select(repo.Get<BasePage>);

It doesn't work with a huge amount of pages. This kind of code fits much better for this issue:

var references = DataFactory.Instance.GetDescendents(RootPageReference);
var pages = DataFactory.Instance.GetItems(references,
LanguageSelector.AutoDetect()).Where(x => !x.IsDeleted).OfType<BasePage>();