Picking with a physics library (OPENGL & BULLET 3D

Question

I'm trying to use bullet physics to draw a ray to hit a game object in the scene so I can select it, i am using the camera matrix to draw a ray and then pick an object in space and then look through a list of game objects and look for the same location.

On Mouse press I have the following code, it seems to be off and only picks the items some times:

    glm::vec4 lRayStart_NDC(
    ((float)lastX / (float)RECT_WIDTH - 0.5f) * 2.0f, 
    ((float)lastY / (float)RECT_HEIGHT - 0.5f) * 2.0f,
    -1.0,
    1.0f
);

glm::vec4 lRayEnd_NDC(
    ((float)lastX / (float)RECT_WIDTH - 0.5f) * 2.0f,
    ((float)lastY / (float)RECT_HEIGHT - 0.5f) * 2.0f,
    0.0,
    1.0f
);


projection = glm::perspective(glm::radians(SceneManagement::getInstance()->MainCamera->GetVOW()), (float)RECT_WIDTH / (float)RECT_HEIGHT, 0.1f, 100.0f);
glm::mat4 InverseProjectionMatrix = glm::inverse(projection);

view = SceneManagement::getInstance()->MainCamera->GetViewMatrix();
glm::mat4 InverseViewMatrix = glm::inverse(view);

glm::vec4 lRayStart_camera = InverseProjectionMatrix * lRayStart_NDC;    
lRayStart_camera /= lRayStart_camera.w;
glm::vec4 lRayStart_world = InverseViewMatrix * lRayStart_camera; 
lRayStart_world /= lRayStart_world.w;
glm::vec4 lRayEnd_camera = InverseProjectionMatrix * lRayEnd_NDC;      
lRayEnd_camera /= lRayEnd_camera.w;
glm::vec4 lRayEnd_world = InverseViewMatrix * lRayEnd_camera;   
lRayEnd_world /= lRayEnd_world.w;

glm::vec3 lRayDir_world(lRayEnd_world - lRayStart_world);
lRayDir_world = glm::normalize(lRayDir_world);

glm::vec3 out_end = SceneManagement::getInstance()->MainCamera->GetCamPosition() + SceneManagement::getInstance()->MainCamera->GetCamFront() * 1000.0f;

btCollisionWorld::ClosestRayResultCallback RayCallback(
    btVector3(SceneManagement::getInstance()->MainCamera->GetCamPosition().x, SceneManagement::getInstance()->MainCamera->GetCamPosition().y, SceneManagement::getInstance()->MainCamera->GetCamPosition().z),
    btVector3(out_end.x, out_end.y, out_end.z)
);

PhysicsManager::getInstance()->dynamicsWorld->rayTest(
    btVector3(SceneManagement::getInstance()->MainCamera->GetCamPosition().x, SceneManagement::getInstance()->MainCamera->GetCamPosition().y, SceneManagement::getInstance()->MainCamera->GetCamPosition().z),
    btVector3(out_end.x, out_end.y, out_end.z),
    RayCallback
);

if (RayCallback.hasHit()) 
{
    btTransform position = RayCallback.m_collisionObject->getInterpolationWorldTransform();
    printf("Collision \n");
    for (int i = 0; i < SceneManagement::getInstance()->gObjects.size(); i++)
    {
        if (SceneManagement::getInstance()->gObjects.at(i)->transform.Position.x == position.getOrigin().getX() &&
            SceneManagement::getInstance()->gObjects.at(i)->transform.Position.y == position.getOrigin().getY() &&
            SceneManagement::getInstance()->gObjects.at(i)->transform.Position.z == position.getOrigin().getZ())
        {
            int select = i;
            SceneManagement::getInstance()->SelectedGameObject = SceneManagement::getInstance()->gObjects.at(select);
            SceneManagement::getInstance()->SelectedGameObject->DisplayInspectorUI();
            return;
        }
    }
}

Answer 1

This check

SceneManagement::getInstance()->gObjects.at(i)->transform.Position.x == position.getOrigin().getX() &&
SceneManagement::getInstance()->gObjects.at(i)->transform.Position.y == position.getOrigin().getY() &&
SceneManagement::getInstance()->gObjects.at(i)->transform.Position.z == position.getOrigin().getZ()

fails due to numerical precision issues.

You should check the "equality" of vectors only up to some precision.

Please, use some distance function and the following check instead of your if() condition:

 const double EPSILON = 1e-4; // experiment with this value
 auto objPos = SceneManagement::getInstance()->gObjects.at(i)->transform.Position;
 bool isWithinRange = distance3D(objPos, position.getOrigin()) < EPSILON;
 if (isWithinRange)
 {
      int select = i;
      ...
 }

The distance3D function is simply the euclidean distance in 3D. The glm::distance function should do, just convert both vectors to glm::vec3 format.