I'm trying to orient an object towards a point in space, using:
m_Z = glm::normalize(target - m_Position);
m_Y = glm::normalize(m_Y - glm::dot(m_Y, m_Z) * m_Z);
m_X = glm::normalize(glm::cross(m_Y, m_Z));
And while the object does "look at" the point in 3D, it seems to rotate around it's own forward vector (m_X) meaning the UP vector is in an incorrect direction. So the object is sometimes looking at the point while being upside down... So it's like a head that's tilted (rotated around local forward vector)
I know how to orient an object towards a point in 2D - that is not what I'm after - I'm looking for a way to correct the upvector in 3d to an extent that the object's "top" is always (more or less) facing up, so I need the object to follow (look at) the point not just left and right, but also up & down.
m_X = left vector of object
m_Y = up vector of object
m_Z = forward vector of object
target = point's world position
m_Position = object's world position
Just re-arrange the order in which you do the calculations:
m_Z = glm::normalize(target - m_Position);
m_X = glm::normalize(glm::cross(up, m_Z));
m_Y = glm::normalize(glm::cross(m_Z, m_X));
Here, up is the perfect up-direction, e.g. (0, 1, 0).
Note that this will fail if up is parallel to target - m_Position as you will have an unconstrained degree of freedom. You would need to add assumptions for that case.