Issue while implementing camera rotation along X and Z axis

Question

I'm implementing a camera which responds to change of mouse position. It's a more question of maths than of coding but I'd like to know how to use it as well.

I have a Camera object which rotates along the Y-axis when the mouse changes its X-position. This works as intended and I can rotate around the cube I'm drawing just fine. Now I would like to implement looking up and down triggered by mouse change vertically but the X and Z-axis are relative to the camera object so I can't just rotate along the X-axis but have to combine the X and Z-axis to do this in a fluid motion.

public class Camera {

    public float moveSpeed = 0.05f;

    private Vector3f position, rotation;
    private float oldMouseX, oldMouseY, newMouseX, newMouseY, mouseSensitivity;

    public Camera () {
        position = new Vector3f(0f, 0f, 0f);
        rotation = new Vector3f(0f, 0f, 0f);

        mouseSensitivity = 0.1f;
        oldMouseX = 0.0f;
        oldMouseY = 0.0f;
        newMouseX = 0.0f;
        newMouseY = 0.0f;
    }

    public Camera (Vector3f pos, Vector3f rot) {
        this.position = pos;
        this.rotation = rot;

        mouseSensitivity = 0.1f;
        oldMouseX = 0.0f;
        oldMouseY = 0.0f;
        newMouseX = 0.0f;
        newMouseY = 0.0f;
    }

    public void setCursor (int x, int y) {
        oldMouseX = x;
        oldMouseY = y;
        newMouseX = x;
        newMouseY = y;
    }

    public Matrix4f getViewMatrix () {
        Matrix4f rotateX = new Matrix4f().rotate(rotation.x * (float)Math.PI / 180f, new Vector3f(1f, 0f, 0f));
        Matrix4f rotateY = new Matrix4f().rotate(rotation.y * (float)Math.PI / 180f, new Vector3f(0f, 1f, 0f));
        Matrix4f rotateZ = new Matrix4f().rotate(rotation.z * (float)Math.PI / 180f, new Vector3f(0f, 0f, 1f));

        Matrix4f rotation = MatrixMath.mul(rotateX, MatrixMath.mul(rotateZ, rotateY));

        Vector3f negPosition = new Vector3f(-position.x, -position.y, -position.z);
        Matrix4f translation = new Matrix4f().translate(negPosition);

        return MatrixMath.mul(translation, rotation);
    }
            public Vector3f getPosition() {
        return position;
    }

    public Vector3f getRotation() {
        return rotation;
    }

    public void update (Window window) {

        if (window.isKeyDown(GLFW.GLFW_KEY_W)) {
            position.x += Math.sin(Math.PI * rotation.y / 180) * -moveSpeed;
            position.z += Math.cos(Math.PI * rotation.y / 180) * moveSpeed;
        }

        if (window.isKeyDown(GLFW.GLFW_KEY_S)) {
            position.x -= Math.sin(Math.PI * rotation.y / 180) * -moveSpeed;
            position.z -= Math.cos(Math.PI * rotation.y / 180) * moveSpeed;
        }

        if (window.isKeyDown(GLFW.GLFW_KEY_D)) {
            position.x += Math.sin(Math.PI * (rotation.y - 90) / 180) * -moveSpeed;
            position.z += Math.cos(Math.PI * (rotation.y - 90) / 180) * moveSpeed;
        }

        if (window.isKeyDown(GLFW.GLFW_KEY_A)) {
            position.x -= Math.sin(Math.PI * (rotation.y - 90) / 180) * -moveSpeed;
            position.z -= Math.cos(Math.PI * (rotation.y - 90) / 180) * moveSpeed;
        }

        if (window.isKeyDown(GLFW.GLFW_KEY_SPACE)) {
            addPosition(0f, moveSpeed, 0f);
        }

        if (window.isKeyDown(GLFW.GLFW_KEY_LEFT_SHIFT)) {
            addPosition(0f, -moveSpeed, 0f);
        }

        newMouseX = (float)window.getMouseX();
        newMouseY = (float)window.getMouseY();

        float dx = newMouseX - oldMouseX;
        float dy = newMouseY - oldMouseY;

        if (window.isMouseButtonDown(GLFW.GLFW_MOUSE_BUTTON_LEFT)) {
            rotation.y += dx * mouseSensitivity;
        }

        //unPos = unPos.rotateAxis(dy * mouseSensitivity, (float)Math.cos(Math.PI * rotation.y / 180), 0f, (float)Math.sin(Math.PI * rotation.y / 180));

//      rotation.x += (float)Math.cos(rotation.y * Math.PI / 180) * (dy * mouseSensitivity);
//      rotation.z += (float)Math.sin(rotation.y * Math.PI / 180) * (dy * mouseSensitivity);

        oldMouseX = newMouseX;
        oldMouseY = newMouseY;
    }
}

I don't think it's necessary to show you my Window class as the functions are quite self-explanitory. As you can see the part at the bottom that I commented out was my approach to solving the problem and at first it seemed to work but the rotating was slightly off.

I expect fluid up and down motion(that is, relative to the camera) but receive a weird rolling motion.

Any help is greatly appreciated!

Answer 1

I fixed my problem. It's strange but I have to multiply the Y-rotation matrix by the X-rotation matrix. That doesn't make sense to me but it works. Thanks for your help!