How can I run multiple instances of a service concurrently in C#?

Question

Currently, I am trying to create a console application in .NET 7, which is able to run multiple instances of a service concurrently.

The Program.cs file looks like this:

private static async Task Main(string[] args)
{
    await Host.CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddHostedService<ConsoleHostedService>();
            services.AddScraperServices();
        })
        .RunConsoleAsync();
}

where AddScraperServices() looks like this:

public static IServiceCollection AddScraperServices(this IServiceCollection services)
{
    services.AddTransient<IScrapingService, ScrapingService>();

    services.AddScoped(provider => new SemaphoreSlim(2));

    return services;
}

and the ConsoleHostedService like this:

internal sealed class ConsoleHostedService : IHostedService
{
    // .. properties and constructor here

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _appLifetime.ApplicationStarted.Register(() =>
        {
            Task.Run(async () =>
            {
                while (!cancellationToken.IsCancellationRequested)
                {
                    await _semaphore.WaitAsync(cancellationToken);

                    try
                    {
                        using (var scope = _serviceProvider.CreateScope())
                        {
                            var scrapingService = scope.ServiceProvider.GetRequiredService<IScrapingService>();

                            await scrapingService.ScrapeAsync();
                        }
                    }
                    catch (Exception ex)
                    {
                        _logger.LogError(ex, "Unhandled exception!");
                    }
                    finally
                    {
                        _semaphore.Release();
                    }
                }
            });
        });

        return Task.CompletedTask;
    }
}

The ScrapingSerivce, which is used in the ConsoleHostedService looks like this:

public class ScrapingService : IScrapingService
{
    // .. properties and constructor here

    public async Task ScrapeAsync()
    {
        await _semaphore.WaitAsync();

        try
        {
            // do stuff
        }
        finally
        {
            _semaphore.Release();
        }
    }
}

As I understand it, Semaphore should now create two instances of ScrapingService running parallel to each other. Spoiler: It does not.

It might be important to mention, that Semaphore should create a new instance, if there is one beeing finished. That means, that I always want the count of instances equal the maximum count of instances.

Does anybody knows, where my problem is? I highly appreciate any kind of help, cheers!

Answer 1

Semaphore should now create two instances of ScrapingService running parallel to each other

Probably just a nitpick and me being pedantic - SemaphoreSlim does not start anything it is just a primitive used for synchronization.

As far as I can see in ConsoleHostedService.StartAsync you are starting only one task which runs an endless loop which awaits every invocation of ScrapeAsync - await scrapingService.ScrapeAsync(); so the next iteration will not start until the current one is finished, so no parallel invocations here.

Personally I would not even bother with the semaphore, and just used some setting to determine the parallelism and passed it to the hosted service (also I would have used BackgroundService over IHostedService, see the docs). Something along these lines:

class MyBackgroundService : BackgroundService
{
    private readonly MyBackgroundServiceSettings _settings;

    public MyBackgroundService(MyBackgroundServiceSettings settings)
    {
        _settings = settings;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var tasks = Enumerable.Range(0, _settings.ParallelCount)
            .Select(_ => Task.Run(async () =>
            {
                while (stoppingToken.IsCancellationRequested)
                {
                    // ...
                }
            }))
            .ToArray();
        await Task.WhenAll(tasks);
    }
}

If for some reason there is a possibility that ScrapingService is resolved somewhere else in the app (though I would suggest to avoid that) and you still want to limit it's concurrent invocations - then I hugely recommend to abstract over the SemaphoreSlim (something like IScrapingServiceLimiter, maybe use ) to avoid the services.AddScoped(provider => new SemaphoreSlim(2)); registration (which looks a bit strange to me, TBH).

P.S.

Also I would recommend to pass the CancellationToken stoppingToken and use it in IScrapingService.ScrapeAsync with corresponding changes to the code.