Rotate camera around a pivot point calculated from mouse click on the screen

Question

My model is shifted to another position on the screen after select pivot point on the 3D model.

I'm trying to use the GL.LookAt to implement the rotation around pivot point feature.

GL.MatrixMode(MatrixMode.Modelview);
Matrix4d viewMatrix = Matrix4d.LookAt(cameraPosition, rotatePoint.point, up);

Matrix4d translateToPivot = Matrix4d.CreateTranslation(-rotatePoint.point);
Matrix4d translateBackFromPivot = Matrix4d.CreateTranslation(rotatePoint.point);

Matrix4d rotationX = Matrix4d.CreateRotationX(MathHelper.DegreesToRadians(-angleY));
Matrix4d rotationY = Matrix4d.CreateRotationY(MathHelper.DegreesToRadians(angleX));

Matrix4d rotationMatrix = rotationY * rotationX;

viewMatrix = translateToPivot * rotationMatrix * translateBackFromPivot * viewMatrix;

GL.LoadMatrix(ref viewMatrix);
SetupViewport();
pco.Render(point_size, ShowOctreeOutline, PointCloudColor, mFrustum);

SetupViewport function:

int w = glControl1.ClientSize.Width;
int h = glControl1.ClientSize.Height;

GL.MatrixMode(MatrixMode.Projection);
GL.LoadIdentity();
float aspect = w / (float)h;
float n = scaling;
float left = -n * 0.5f, right = n * 0.5f, down = -n * 0.5f / aspect, up = n * 0.5f / aspect;

if (w <= h)
{
    GL.Ortho(-1, 1, -aspect, aspect, -1, 1);
}
else
{
    GL.Ortho(left, right, down, up, -10.0f, 10.0f);
}
GL.Viewport(0, 0, w, h);

Answer 1

In my code, I didn't save the previous state and then multiply that previous state to the transformation I did earlier.

The problem is I keep creating a new model matrix base on the origin state so that make my model shifts when I choose a new pivot base on the NEW STATE but meanwhile reset the state to origin. (Example code):

// Global variable
Matrix4d prevModelMatrix = Matrix4d.Identity;
Matrix4d modelMatrix = Matrix4d.Identity;

function Render() {
  ...
  GL.LoadMatrix(modelMatrix);
  ...
  Transformation... (Use offset instead of using new rotate and translate value to avoid accumulating)
  For example: 
    - GL.Rotate(offsetAngleX, 1,0,0);
}

// Reset offsets to 0 to avoid Render() function still use the offset to transform the scene
function MouseUp() {
  offsetAngleX = 0;
  ...
}