Async Service processes entity multiple times

Question

I have a very strange behaviour of my Async service. The story is: There's a plugin, which fires on Lead Create. The purpose of the plugin itself is to create custom enumeration of Leads. The plugin gets the last numer from the field in tha Autonumbering entity which keeps numbers. Then the plugin increments the Autonumbering entity' number field by 1 and assigns obtained number to Lead.

The problem is following: When I run mass-creation of leads (crash-test for numbering) e.g. 400, and Autonumbering counter starts from 0, when all Leads are processed my Autonumbering counter ends with the value of ~770, what is much more than estimated 400.

I found by experience that Async service processes same leads multiple times. For some only 1 time, for others it is 2-5 times.

Why this happens?

Here's my code:

public void Execute(IServiceProvider serviceProvider)
{
    Entity target = ((Entity)context.InputParameters["Target"]);
    target["new_id"] = GetCurrentNumber(service, LEAD_AUTONUMBER);
    service.Update(target);
    return;
}

public int GetCurrentNumber(IOrganizationService service, Guid EntityType)
{
    lock (_locker)
    {
        Entity record = service.Retrieve("new_autonumbering", EntityType, new ColumnSet("new_nextnumber"));
        record["new_nextnumber"] = int.Parse(record["new_nextnumber"].ToString()) + 1;
        service.Update(record);

        return int.Parse(record["new_nextnumber"].ToString());
    }
}

UPDATE 1: First my context-factory-service variables were declared in the class so they could be used one instance for multiple threads.

public class IdAssignerPlugin : IPlugin
{
    private static      IPluginExecutionContext context;
    private static      IOrganizationServiceFactory factory;
    private static      IOrganizationService service;

    public void Execute(IServiceProvider serviceProvider)
    {
        context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
        factory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
        service = factory.CreateOrganizationService(null);
        [...]
    }
}

After the comment of @HenkvanBoeijen I realised that this isn't safe way, so I moved all declarations into Execute() method.

public class IdAssignerPlugin : IPlugin
{
    public void Execute(IServiceProvider serviceProvider)
    {
        IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));;
        IOrganizationServiceFactory factory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));;
        IOrganizationService service = factory.CreateOrganizationService(null);;

        [...]
    }
}

But this not saved me from multiple processing, although the processing now goes much rapidly.

UPDATE 2: In System Jobs I have also noticed that after 11 operations with status Retry Count = 0 the rest operations have Retry Count = 1, and after 16 it is Retry Count = 2, etc.

(in this test I created 20 leads programmaticaly, and after assigment the counter shows me last number = 33, and if I summarize all retry count values it comes out with 33, which is similiar to last number in Autonumbering)

Answer 1

I found the problem. After 11 tries on all following tasks CRM had been showing plugin error (It was Generic SQL Error with no any additional info, and I suppose it may be caused by overload, some kind of SQL Timeout Error).

Event execution pipeline for crm is following:

1. Event happened.
2. Event listener cathes event and sends it to the handler based on following parameters sync-async & pre-operation - post-operation (async - post-operation in my case)
3. Then event goes into Async Queue Agent which decides when to execute the plugin.
4. Async Queue Agent runs related to this event plugin.
5. Plugin does his work and then returns 0 (for e.g.) when succeeded or 1 when failed.
6. If 0, Async Queue Agent closes current pipeline with the status of Succeeded and sends notification to CRM core.

The error presumably arose after update of Autonumbering inside the code (step 5) entity but before completion the task with status succeeded (step 6).

So due to this error CRM run the task with the same InputParameters again.

My CRM server is not very overloaded so I come out with the following workaround:

I extrapolate my lock() statement on the whole Execute() method and move Update Entity request to the end of the method.

Everything goes fine. The drawback is that this way turns back my plugin into (almost) old-good synchronous, but as I said my server is not so overloaded to not afford this issue.

I post my code for historical reasons:

public class IdAssignerPlugin : IPlugin
{
    public const string AUTONUMBERING_ENTITY = "new_autonumber";
    public static Guid LEAD_AUTONUMBER = 
new Guid("yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy");

    static readonly object _locker = new object();

    public void Execute(IServiceProvider serviceProvider)
    {
        var context = (IPluginExecutionContext)serviceProvider.GetService(
typeof(IPluginExecutionContext));
        var factory = (IOrganizationServiceFactory)serviceProvider.GetService(
typeof(IOrganizationServiceFactory));
        var service = factory.CreateOrganizationService(null);

        if (PreventRecursiveCall(context))
        return;

        lock (_locker)
        {
            if (context.InputParameters.Contains("Target") && 
context.InputParameters["Target"] is Entity)
            {
                Entity autoNumber;
                Entity target = ((Entity)context.InputParameters["Target"]);
                if (target.LogicalName.Equals("lead",
 StringComparison.InvariantCultureIgnoreCase))
                {
                    autoNumber = GetCurrentNumber(service, LEAD_AUTONUMBER);
                    target["new_id"] = autoNumber["new_nextnumber"];
                }
                service.Update(autoNumber);
                service.Update(target);
            }
        }
        return;
    }

    public int GetCurrentNumber(IOrganizationService service, Guid EntityType)
    {
        Entity record = 
service.Retrieve(AUTONUMBERING_ENTITY, EntityType, new ColumnSet("new_nextnumber"));
        record["new_nextnumber"] = int.Parse(record["new_nextnumber"].ToString()) + 1;
        return record;
    }

    protected virtual bool PreventRecursiveCall(IPluginExecutionContext context)
    {
        if (context.SharedVariables.Contains("Fired")) return true;
        context.SharedVariables.Add("Fired", 1);
        return false;
    }
}