Problem with SortedDictionary in C# 11 with .NET 4.8 / Globalization

Question

A web site uses the current culture of the user for displaying data. A problem occured when a user from Vietnam accessed the data. Can someone explain why this problems occurs? Code to reproduce the error:

static void SortedDicProblem(Action<string> Write)
{
    Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");

    const string key = "KH";

    SortedDictionary<string, string> laCountries = new()
    {
        { key, "Kambodscha " },
        { "KI", "Kiribati " },
        { "KM", "Komoren " },
    };

    foreach (CultureInfo loCult in System.Globalization.CultureInfo.GetCultures(CultureTypes.InstalledWin32Cultures))
    {
        Test(loCult.Name);
    }

    void Test(string vsCulture)
    {
        Thread.CurrentThread.CurrentCulture = new CultureInfo(vsCulture);

        if (laCountries.TryGetValue(key, out string lsName))
        {
            Write($"{vsCulture}");
        }
        else
        {
            Write($"<span class='error'>{vsCulture}, Key not found: {key}, {Thread.CurrentThread.CurrentCulture.EnglishName}</span>");
        }
    }
}

We'll know how to solve it, but what's the reason for this error? A Solution to avoid the problem: Instantiate the SortedDictionary with a StringComparer, like:

new SortedDictionary<string,string>(StringComparer.Ordinal)

Answer 1

Here is the issue (occurs on .NET 4.8, not on .NET 5+):

public static void Main()
{
    var usComparer = StringComparer.Create(new CultureInfo("en-US"), ignoreCase: false);
    var veComparer = StringComparer.Create(new CultureInfo("vi-VN"), ignoreCase: false);

    Console.WriteLine(usComparer.Compare("KH", "KM")); // Prints -1
    Console.WriteLine(veComparer.Compare("KH", "KM")); // Prints  1
}

In English-US, "KH" is considered to be before "KM". In Vietnamese, "KH" is considered to be after "KM".

Because you have inserted the items into the dictionary using one comparer, and then tried to retrieve them using a different comparer which gives a different sort order, it all falls apart.

As to why it's different on .NET 5+ - well that's a different question. I would imagine that it's to do with the fact that .NET core uses ICU for string comparisons and .NET 4.8 uses NLS.

---

You can make the issue also occur on .NET 5+ by forcing the use of NLS rather than ICU by adding the following to the project file and recompiling:

<ItemGroup>
  <RuntimeHostConfigurationOption Include="System.Globalization.UseNls" Value="true" />
</ItemGroup>

---

ADDENDUM

I couldn't work out why this was failing, given that the implementation of SortedDictionary caches the comparer. Surely it should give the same results even when you change the locale?

Well it turns out that Comparer<string>.Default actually changes its behaviour when you change locale, even if it's been cached!

Consider this code, which demonstrates:

var comparer = Comparer<string>.Default; // "Cache" the comparer.
Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");
Console.WriteLine(comparer.Compare("KH", "KM"));                  // Prints -1
Thread.CurrentThread.CurrentCulture = new CultureInfo("vi-VN");
Console.WriteLine(comparer.Compare("KH", "KM"));                  // Prints 1

That explains why the search goes wrong even though the comparer was cached.

But what explains the behaviour of the cached comparer? It's quite simple. The cached comparer is of type GenericComparer<T>, the implementation of which is:

    public override int Compare(T x, T y)
    {
        if (x != null)
        {
            if (y != null) return x.CompareTo(y);
            return 1;
        }
        if (y != null) return -1;
        return 0;
    }

So since T is string, this is effectively just calling the string.CompareTo() method, which will be using the current locale.