Problems rotating opengl camera

Question

I cannot understand the math behind this problem, I am trying to create an FPS camera where I can look freely with my mouse input.

I am trying to rotate and position my lookat point with 180 degrees of freedom. I understand the easier solution is to glRotate the world to fit my perspective, but I do not want this approach. I am fairly unfamiliar with the trigonometry involved here and cannot figure out how to solve this problem the way I want to...

here is my attempt to do this so far...

code to get mouse coordinates relative to the center of the window, then process it in my camera object

#define DEG2RAD(a) (a * (M_PI / 180.0f))//convert to radians
static void glutPassiveMotionHandler(int x, int y) {
    glf centerX = WinWidth / 2; glf centerY = WinHeight / 2;//get windows origin point
    f speed = 0.2f;
    f oldX = mouseX;  f oldY = mouseY;

    mouseX = DEG2RAD(-((x - centerX)));//get distance from 0 and convert to radians
    mouseY = DEG2RAD(-((y - centerY)));//get distance from 0 and convert to radians

    f diffX = mouseX - oldX; f diffY = mouseY - oldY;//get difference from last frame to this frame

    if (mouseX != 0 || mouseY != 0) {
        mainCamera->Rotate(diffX, diffY);
    }

Code to rotate the camera

void Camera::Rotate(f angleX, f angleY) {
    Camera::refrence = Vector3D::NormalizeVector(Camera::refrence * cos(angleX)) + (Camera::upVector * sin(angleY));//rot up
    Camera::refrence = Vector3D::NormalizeVector((Camera::refrence * cos(angleY)) - (Camera::rightVector * sin(angleX)));//rot side to side
};

Camera::refrence is our lookat point, processing the lookat point is handled as follows

void Camera::LookAt(void) {
    gluLookAt(
        Camera::position.x, Camera::position.y, Camera::position.z,
        Camera::refrence.x, Camera::refrence.y, Camera::refrence.z,
        Camera::upVector.x, Camera::upVector.y, Camera::upVector.z
    );
};

Answer 1

The camera is defined by a position point (position) a target point (refrence) and a up-vector upVector. If you want to change the orientation of the camera, then you've to rotate the direction vector from the position (position) to the target (refrence) rather then the target point by a Rotation matrix.

Note, since the 2 angles are angles which should change an already rotated view, you've to use a rotation matrix, to rotate the vectors which point in an arbitrary direction.

Write a function which set 3x3 rotation matrix around an arbitrary axis:

void RotateMat(float m[], float angle_radians, float x, float y, float z)
{
    float c = cos(angle_radians);
    float s = sin(angle_radians);

    m[0] = x*x*(1.0f-c)+c;   m[1] = x*y*(1.0f-c)-z*s; m[2] = x*z*(1.0f-c)+y*s;
    m[3] = y*x*(1.0f-c)+z*s; m[4] = y*y*(1.0f-c)+c;   m[5] = y*z*(1.0f-c)-x*s;
    m[6] = z*x*(1.0f-c)-y*s; m[7] = z*y*(1.0f-c)+x*s; m[8] = z*z*(1.0f-c)+c };
}

Write a function which rotates a 3 dimensional vector by the matrix:

Vector3D Rotate(float m[], const Vector3D &v)
{
  Vector3D rv;
  rv.x = m[0] * v.x + m[3] * v.y + m[6] * v.z;
  rv.y = m[1] * v.x + m[4] * v.y + m[7] * v.z;
  rv.z = m[2] * v.x + m[5] * v.y + m[8] * v.z;
  return rv;
}

Calculate the vector form the position to the target:

Vector3D los = Vector3D(refrence.x - position.x, refrence.y - position.y, refrence.z - position.z);

Rotate all the vectors around the z axis of the world by angleX:

float rotX[9];
RotateMat(rotX, angleX, Vector3D(0, 0, 1));

los = Rotate(rotX, los);
upVector = Rotate(rotX, upVector);

Rotate all the vectors around the current y axis of the view by angleY:

float rotY[9];
RotateMat(rotY, angleY, Vector3D(los.x, los.y, 0.0));

los = Rotate(rotY, los);
upVector = Rotate(rotY, upVector);

Calculate the new target point:

refrence = Vector3D(position.x + los.x, position.y + los.y, position.z + los.z);