How to distinguish the nearest bluetooth deivce in a small area accurately?

In my app,I use ble(Bluetooth Low Energy) to scan and connect to a nearest bluetooth device(There exists two similar bluetooth devices nearby).I use RSSI to make sure which is nearest and in most cases,it works fine.But I find it not 100% correct when the distances are short.During my test,one is 2 meters away from me and the other is 3 meters,and the RSSI of the farther one comes to be bigger,about 1 in 10 times.Is there any better idea to replace RSSI?

Solution

Your problem is very well known and it appears in any localization algorithm using ble beacons. Even if two devices are very close together, they may have different RSSI value due to the Fast fading effect.

The Fast fading originates due to effects of constructive and destructive interference patterns which is caused due to multipath.

To mitigate this problem, you can :

Compare the RSSI during a longer time. Especially if things are moving around, the radio-waves may interfere in a different way. If your receiver is a smartphone for example, the user is not static and a few more RSSI sample will give you a more accurate results.
Add spacial diversity. This can be done by adding another chip with another antenna that will also advertise. If the two antenna are not at the same place, you will have more RSSI data, coming from different path that will interfere in a different way. By doing the mean of the two value you should have better result (ideally combine with a longer acquisition time). But of course it is only possible if you are designing the hardware of the advertising device. Note that this can also allow your reciever to catch more adv for the same time-frame.
Frequency diversity. Make sure your advertiser is configured to use the 3 adv channels.

And of course if the two distances are very different the slow fading will be greater than any fast fading effect and you should not have any trouble.