Suppose there is a replica set of three nodes: 1 primary P and 2 secondaries S1 and S2. Suppose also that P and S1 are located in one data center DC1 and S2 in another data center DC2.
Suppose a client is located in DC1 (i.e. in the same data center as P and S1) and S1 is down. Where will the client queries go ? P or S2 ?
secondaryPreferred will choose the secondary, even if it's further away (I'm assuming that S2 has a lower priority because it probably shouldn't be primary, so in your scenario S2 is the secondary and that it's further away in the latency sense).
You can instead use tag sets for data-center awareness / geographically distributed members or the nearest read-preference, which will select the server that is closer, irrespective of whether it's a primary or secondary.