Is there a performance hit when I use SortedDictionary<> rather than Dictionary<> taking into consideration that I need a sorted Dictionary, so if I used a Dictionary, I would have to manually sort it. Which would be faster?
SortedDictionary<> or (Dictionary<> + Manual Sort).
Inserts into a SortedDictionary are O(log(n)), so inserting n items is O(n log(n)). In contrast, inserts into a Dictionary are O(1), so inserting n items is O(n), but you'll need to sort the items before use, which is O(n log(n)). Based on that, there isn't much difference, but Dictionary has the overhead of hashing, while the SortedDictionary might have worse memory locality since it is probably implemented as a linked structure.
SortedDictionary also places a limit on the key types you can use since it needs to sort by the key, while Dictionary does not since it uses hashing.
Really, which is better depends on your access patterns, so the best thing to do is to measure the performance of both for your use case.