Custom IEqualityComparer to get distinct objects from two lists faster than using .Where() + .Any()

Question

so I've got two lists of objects, objects have multiple fields, but I'd like to distinct them basing on only two of them.

To give you the picture, object KeyAndValue consists of fields Key and Tag, so:

list1 = { obj1(key=1,tag=A), obj2(key=2,tag=A) }
list2 = { obj3(key=1,tag=A), obj4(key=2,tag=B) }

I'm currently using:

list1.Where(x => !list2.Any(y => y.key == x.key)).ToList();

The correct result is: obj1, obj2 and obj4 as obj3 has the same key and tag as obj1

What I'm trying to accomplish is to speed this process up as with a lot of objects it takes much too long. I found that custom IEqualityComparer could help here, so I've written my own basing on MS Specification It looks like this:

class KeyComparer : IEqualityComparer<KeyAndValue>
{
    public bool Equals(KeyAndValue x, KeyAndValue y)
    {
        if (Object.ReferenceEquals(x, y))
            return true;
        if (Object.ReferenceEquals(x, null) || Object.ReferenceEquals(y, null))
            return false;
        return x.key == y.key && x.tag == y.tag;
    }
    public int GetHashCode(KeyAndValue keyAndValue)
    {
        if (Object.ReferenceEquals(keyAndValue, null))
            return 0;
        int hashKeyAndValueKey = keyAndValue.key == null ? 0 : keyAndValue.key.GetHashCode();
        int hashKeyAndValueTag = keyAndValue.tag == null ? 0 : keyAndValue.tag.GetHashCode();
        return hashKeyAndValueKey ^ hashKeyAndValueTag;
    }
}

And I use it like this:

list1.Except(list2, new KeyComparer()).ToList();

Unfortunately it does only remove duplicates from list2. It seems that it does not even touch list1 and I do not know if it's the fault of my custom comparer, the way I use it or maybe I should use another method. I've been looking through other questions but could not find a working answer (or at least one that I'd actually know how to implement properly).

Answer 1

As Stefan Steinegger (whom I thank grately for the effort and time spent) mentioned in the first comment that neither of my methods return obj4 I found an issue and decided to implement totally different approach. Now my KeyAndValue class has also an int Hash field, when a constructor is called Hash field is filled with key.GetHashCode() ^ tag.GetHashCode(). It simplified the comparison as now I'm first combining list1 with list2 and then sending it through: CombinedList.GroupBy(x => x.Hash).Select(y => y.First()).ToList(); Results seem to be correct :)