I have a NavMeshAgent that I am using SetDestination() with, but the agent always tries to get as close to the destination as possible, even the location is unreachable.
Try to use NavMeshPathStatus.PathComplete to check if the path ends at the destination.
Here's Unity answers for a similar question https://answers.unity.com/questions/1197626/navmesh-how-to-check-if-full-path-available-c.html