Let's say that I have Dictionary<string,IEnumerable<int>> and:
Task1 tried to replace value under some key.
Task2 loops through elements that are in value under the same key.
Is this thread safe? I assume it should be if I first get the value and then loop though it in Task2. Task2 in this case should either get old value or new value, but I think it should never generate any thread exceptions.
When discussing thread safety, you need to be very precise and specific. Short answer "no". Longer answer "it could be, if you work at it, but it would be better to redesign this a bit".
In terms of the dictionary, if we assume that you're mutating the dictionary: we need to consider a get vs a set, competing. That is not thread safe unless you synchronize over the individual operations - you could absolutely see exceptions when accessing a dictionary concurrently if at least one access is a mutation. But frankly: it might make more sense to use ConcurrentDictionary<,> instead.
Separately, we need to consider enumeration of the sequence, IEnumerable<int>. Now; IEnumerable<int> does not guarantee to be repeatable, and the fact that it is in a lookup means you're probably planning on enumerating it multiple times, so: that's not ideal. Many types that implement IEnumerable<T> are repeatable, but it isn't an API guarantee. Further: many of those types are themselves mutable, and therefore not necessarily thread-safe vs changes to those types.
So: if it was me, I might consider using ConcurrentDictionary<string, ImmutableArray<int>> or ConcurrentDictionary<string, ReadOnlyMemory<int>>, and then I'd feel confident that it is safe...
...as long as no-one uses reflection or unsafe code, in which case all bets are off!