Can I remove items from a ConcurrentDictionary from within an enumeration loop of that dictionary?

Question

So for example:

ConcurrentDictionary<string,Payload> itemCache = GetItems();

foreach(KeyValuePair<string,Payload> kvPair in itemCache)
{
    if(TestItemExpiry(kvPair.Value))
    {   // Remove expired item.
        itemCache.TryRemove(kvPair.Key, out Payload removedItem);
    }
}

Obviously with an ordinary Dictionary<K,V> this will throw an exception, because removing items changes the dictionary's internal state during the life of the enumeration. It's my understanding that this is not the case for a ConcurrentDictionary, as the provided IEnumerable handles internal state changing. Am I understanding this right? Is there a better pattern to use?

Answer 1

It's strange to me that you've now received two answers that seem to confirm you can't do this. I just tested it myself and it worked fine without throwing any exception.

Below is the code I used to test the behavior, followed by an excerpt of the output (around when I pressed 'C' to clear the dictionary in a foreach and S immediately afterwards to stop the background threads). Notice that I put a pretty substantial amount of stress on this ConcurrentDictionary: 16 threading timers each attempting to add an item roughly every 15 milliseconds.

It seems to me this class is quite robust, and worth your attention if you're working in a multithreaded scenario.

Code

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;

namespace ConcurrencySandbox {
    class Program {
        private const int NumConcurrentThreads = 16;
        private const int TimerInterval = 15;

        private static ConcurrentDictionary<int, int> _dictionary;
        private static WaitHandle[] _timerReadyEvents;
        private static Timer[] _timers;
        private static volatile bool _timersRunning;

        [ThreadStatic()]
        private static Random _random;
        private static Random GetRandom() {
            return _random ?? (_random = new Random());
        }

        static Program() {
            _dictionary = new ConcurrentDictionary<int, int>();
            _timerReadyEvents = new WaitHandle[NumConcurrentThreads];
            _timers = new Timer[NumConcurrentThreads];

            for (int i = 0; i < _timerReadyEvents.Length; ++i)
                _timerReadyEvents[i] = new ManualResetEvent(true);

            for (int i = 0; i < _timers.Length; ++i)
                _timers[i] = new Timer(RunTimer, _timerReadyEvents[i], Timeout.Infinite, Timeout.Infinite);

            _timersRunning = false;
        }

        static void Main(string[] args) {
            Console.Write("Press Enter to begin. Then press S to start/stop the timers, C to clear the dictionary, or Esc to quit.");
            Console.ReadLine();

            StartTimers();

            ConsoleKey keyPressed;
            do {
                keyPressed = Console.ReadKey().Key;
                switch (keyPressed) {
                    case ConsoleKey.S:
                        if (_timersRunning)
                            StopTimers(false);
                        else
                            StartTimers();

                        break;
                    case ConsoleKey.C:
                        Console.WriteLine("COUNT: {0}", _dictionary.Count);
                        foreach (var entry in _dictionary) {
                            int removedValue;
                            bool removed = _dictionary.TryRemove(entry.Key, out removedValue);
                        }
                        Console.WriteLine("COUNT: {0}", _dictionary.Count);

                        break;
                }

            } while (keyPressed != ConsoleKey.Escape);

            StopTimers(true);
        }

        static void StartTimers() {
            foreach (var timer in _timers)
                timer.Change(0, TimerInterval);

            _timersRunning = true;
        }

        static void StopTimers(bool waitForCompletion) {
            foreach (var timer in _timers)
                timer.Change(Timeout.Infinite, Timeout.Infinite);

            if (waitForCompletion) {
                WaitHandle.WaitAll(_timerReadyEvents);
            }

            _timersRunning = false;
        }

        static void RunTimer(object state) {
            var readyEvent = state as ManualResetEvent;
            if (readyEvent == null)
                return;

            try {
                readyEvent.Reset();

                var r = GetRandom();
                var entry = new KeyValuePair<int, int>(r.Next(), r.Next());
                if (_dictionary.TryAdd(entry.Key, entry.Value))
                    Console.WriteLine("Added entry: {0} - {1}", entry.Key, entry.Value);
                else
                    Console.WriteLine("Unable to add entry: {0}", entry.Key);

            } finally {
                readyEvent.Set();
            }
        }
    }
}

Output (excerpt)

cAdded entry: 108011126 - 154069760   // <- pressed 'C'
Added entry: 245485808 - 1120608841
Added entry: 1285316085 - 656282422
Added entry: 1187997037 - 2096690006
Added entry: 1919684529 - 1012768429
Added entry: 1542690647 - 596573150
Added entry: 826218346 - 1115470462
Added entry: 1761075038 - 1913145460
Added entry: 457562817 - 669092760
COUNT: 2232                           // <- foreach loop begins
COUNT: 0                              // <- foreach loop ends
Added entry: 205679371 - 1891358222
Added entry: 32206560 - 306601210
Added entry: 1900476106 - 675997119
Added entry: 847548291 - 1875566386
Added entry: 808794556 - 1247784736
Added entry: 808272028 - 415012846
Added entry: 327837520 - 1373245916
Added entry: 1992836845 - 529422959
Added entry: 326453626 - 1243945958
Added entry: 1940746309 - 1892917475

Also note that, based on the console output, it looks like the foreach loop locked out the other threads that were trying to add values to the dictionary. (I could be wrong, but otherwise I would've guessed you would've seen a bunch of "Added entry" lines between the "COUNT" lines.)