Does ManualResetEventSlim signaling degrades performance?

Question

I am using ManualResetEventSlim to have signaling mechanism in my application and It works great if requests/sec are 100. As I increase request/sec, it gets worse.

Example:

100 Requests/sec -> 90% transaction done in 250 ms and Throughput (Success request/sec) is 134.

150 Requests/sec -> 90% transaction done in 34067 ms and Throughput (Success request/sec) is 2.2.

I use ConcurrentDictionary as give below:

// <key, (responseString,ManualResetEventSlim) >
private static ConcurrentDictionary<string, (string, ManualResetEventSlim)> EventsDict = new ConcurrentDictionary<string, (string, ManualResetEventSlim)>();

Below given process describes need for ManualResetEventSlim (Api Solution 1 and Api Solution 2 are completely :

1. Api Solution 1 (REST Api) received a request, it added an element (null, ManualResetEventSlim) in ConcurrentDictionary against a key and called thirdparty service (SOAP) using async/await. Thirdparty soap api returned acknowledgement response but actual response is pending. After getting acknowledgement response, it goes to ManualResetEventSlim.wait

2. Once thirdparty processed the request, it calls Api Solution 2 (SOAP) using exposed method and sends actual response. Api solution 2 sends response to Api Solution 1 (REST Api) by making http request and then inserts data to database for auditlog.

3. Api Solution 1 will get key from response string and update response string in ConcurrentDictionary and set signal.

4. Api Solution 1 disposes ManualResetEventSlim object before returning response to client.

Answer 1

I think, you should be able to get rid of the blocking code by replacing (string, ManualResetEventSlim) with TaskCompletionSource<string>:

In Solution 1, you would do something along this:

TaskCompletionSource<string> tcs = new TaskCompletionSource<string>()
EventsDict.AddOrUpdate( key, tcs );
await KickOffSolution2ThirdParty( /*...*/ );
string result = await tcs.Task; // <-- now not blocking any thread anymore

And the counterpart:

void CallbackFromSolution2( string key, string result )
{
     if( EventsDict.TryRemove(key, out TaskCompletionSource<string> tcs )
     {
         tcs.SetResult(result);
     }
}

This is of course only a coarse outline of the idea. But hopefully enough to make my line of thought understandable. I cannot test this right now, so any improvements/corrections welcome.