Call Web API for 30K times within one hour of time period

Question

I need to call Web API(Hosted on different network) from windows application from users machine for 30,000 times within one hour of time span.

I tried Multithreading to achieve the same but it is not working(giving system out of memory exceprion).

I used TreadPool as below

private static object threadLock = new object();
public delegate void BarDelegate();

int ThreadCount = dtExcel.Rows.Count;

private void button2_Click(object sender, EventArgs e)
{
    for (int i = 0; i < ThreadCount - 1; i++)
    {
        ThreadPool.QueueUserWorkItem(output => CallAPI());
    }
}

public void CallAPI()
{
    string branchCode = "",
        para1 = dtExcel.Rows[progressBar.Value]["para1"].ToString(),
        para2 = "324",
        para3 = "Test",
        para4 = dtExcel.Rows[progressBar.Value]["para4"].ToString();
    //Console.WriteLine(Thread.CurrentThread.Name + ": " + progressBar.Value);
    var service = new APIService();
    var resp = service.CallAPIService(para1, para2, para3, para4, para5);
    if (resp.IsSuccess == true)
    {
        DataGridViewRow dtGrdVwR = dataGrid.Rows[progressBar.Value];
        dtGrdVwR.Cells[3].Value = "Success";
    }
    else
    {
        DataGridViewRow dtGrdVwR = dataGrid.Rows[progressBar.Value];
        dtGrdVwR.Cells[3].Value = "Failed: "+ resp.Message;
    }
    try
    {
        this.Invoke(new BarDelegate(UpdateBar));
    }
    catch
    {

    }
}

private void UpdateBar()
{

    lblEndTime.Text = DateTime.Now.ToString();
    progressBar.Value++;
    if (progressBar.Value == progressBar.Maximum)
    {
        // We are finished and the progress bar is full.
    }
}

Here dtExcel has 30,000 records(Uploaded by user from excel) which needs to processed within one hour and update the status of executed record in respective row in dataGrid.

The API call is made over network where single call takes approximate 1 to 2 seconds to execute.

service.CallAPIService(para1, para2, para3, para4, para5);

The above method internally performs heavy task like request encryption and digital signature and response decryption and digital signature verification.

Please help me with the best way possible where i can perform the task within time period and without getting SystemOutOfmemoryException.

Thanks in Advance.

Answer 1

Right now your code is horribly broken because of the race condition accessing progressBar.Value. It's pointless to discuss any other issues, because you are going to totally reorganize your code to fix the race condition, rendering other comments obsolete.

Fix it so that you don't have N threads all trying to process item #1, and then ask a new question with your new code.