dataframe logical_and works fine with equals and don't work with not equals

Question

Please help me understand why the "not equal" condition doesn't work properly.

>>>d = {'a' : [1, 2, 3, 3, 1, 4],
>>>     'b' : [4, 3, 2, 1, 2, 2]}
>>>df = pd.DataFrame(d)
    a   b
0   1   4
1   2   3
2   3   2
3   3   1
4   1   2
5   4   2

We get the correct result if I use the equal condition with logical_and:

>>>df[np.logical_and(df['a']==3, df['b']==2)]
    a   b
2   3   2

But if we change the condition to not equal it stops working correctly:

>>>df[np.logical_and(df['a']!=3, df['b']!=2)]
    a   b
0   1   4
1   2   3

This works like the condition OR instead of AND.

But it works fine again if we use ~ before np.logical_and

>>>df[~np.logical_and(df['a']==3, df['b']==2)]
    a   b
0   1   4
1   2   3
3   3   1
4   1   2
5   4   2

What should I know about logical conditions to avoid failure?

Answer 1

I think you should understand De Morgan's Laws:

not (A or B) == (not A) and (not B)

not (A and B) == (not A) or (not B)

This is simply propositional logic, and has nothing to do with Python itself.

We can verify it ourselves with a truth table. If we make a truth table for A and B, we see:

 |A|a|
-+-+-+
B|T|F|
-+-+-+
b|F|F|
-+-+-+

Here A denotes that A is true, and a denotes that A is false (same for B). We denote T for true and F for false. Now the opposite table is thus:

 |A|a|
-+-+-+
B|F|T|
-+-+-+
b|T|T|
-+-+-+

But if we construct a truth table for (not A) and (not B) we obtain:

 |A|a|
-+-+-+
B|F|F|
-+-+-+
b|F|T|
-+-+-+

So the two are not equivalent.

See it like this: if the condition is:

A must be 5 and B must be 3.

Then the opposite is not A must not be 5 and B must not be 3. Since now a case where A is 5 and B is 2 does not satisfies our first condition, but neither does it satisfies our (false) second claim. The opposite is:

A must not be 5 or B must not be 3 (opposite)

Since from the moment one of the two is not 5 or 3 it is sufficient.