what is the faster and best way to make a few million SOAP requests and save the results into SqlDb

Question

I have a million records in my table. I want to call a soap service and i need to do process in all the records in less than one hour. and besides i should update my table , insert the requests and responses in my other tables. but the code below works on less than 10 records every time i run my app.

I know My code is wrong,, I want to know what is the best way to do it.

 static async Task Send( )
    {
        var results = new ConcurrentDictionary<string, int>();

        using (AppDbContext entities = new AppDbContext())
        {
            var List = entities.Request.Where(x => x.State == RequestState.InitialState).ToList();

            Parallel.ForEach(Enumerable.Range(0, List.Count), async index =>
            {
                var selected = List.FirstOrDefault();
                 List.Remove( selected );
                var res1 = await DoAsyncJob1(selected);  ///await
               // var res = CallService(selected);
                var res2 = await DoAsyncJob2(selected);  ///await
                var res3 = await DoAsyncJob3(selected); ///await

               // var responses = await Task.WhenAll(DoAsyncJob1, DoAsyncJob2, DoAsyncJob3);

                // results.TryAdd(index.ToString(), res);
            }); 

        }

    }

    static async Task<int> DoAsyncJob1(Request item)
    { 
        using (AppDbContext entities = new AppDbContext())
        {
            var bReq = new BankRequest();
            bReq.Amount = Convert.ToDecimal(item.Amount);
            bReq.CreatedAt = DateTime.Now; 
            bReq.DIBAN = item.DIBAN;
            bReq.SIBAN = item.SIBAN; 
            entities.BankRequest.Add(bReq);
            entities.SaveChanges();
        }
        return item.Id;
    }


static async Task<int> DoAsyncJob2(Request item)
    { 
        using (AppDbContext entities = new AppDbContext())
        {

        }
        return item.Id;
    }


static async Task<int> DoAsyncJob3(Request item)
    { 
        using (AppDbContext entities = new AppDbContext())
        {

        }
        return item.Id;
    }

Maybe the below lines are wrong :

  var selected = List.FirstOrDefault();
  List.Remove( selected );

Thanks in advance..

Answer 1

First, it is a bad practice to use async-await within Parallel.For - you introduce only more load to the task scheduler and more overhead.

Second, you are right:

var selected = List.FirstOrDefault();
List.Remove( selected );

is very, very wrong. Your code will behave in a totally unpredictable way, due to the race conditions.