Rotating a globe in Unity while keeping it the right way up

Question

Here is the problem:

I have a globe in unity, and by right-clicking and dragging the mouse, it can be rotated both around the equator and the parallel axis horizontal to the camera (in this case the absolute x-Axis, keeping the poles on th YZ-plane), and this works fine, no problems.

The other way of rotating the globe is to click on labels, which automatically rotates the globe to center the label on the camera. The globe starts centered on global coordinates 0,0. This works, however the further away the label is from the prime meridian (so the further east or west from 0 lat.), the further the poles rotate away from their intended axis (instead of remaining along the YZ-plane). Does anyone have any clues on how to achieve this?

Here is the relevant code:

public class LabelBehaviour : MonoBehaviour
{
    [...]
    public void HomeIn()
    {
        globe.transform.rotation = Quaternion.identity;
        Vector3 fromDirection = transform.position;
        Vector3 toDirection = mainCamera.transform.position;
        InputController.Instance.rotateTo = Quaternion.FromToRotation(fromDirection, toDirection);
    }
}

//InputController class is attached to the globe object
public class InputController : MonoBehaviour {
    [...]
    void Update()
    {
        if(Quaternion.Angle(rotateTo,transform.rotation) > .1f)
            //rotate towards position in 1.5s
            rotateTime = Mathf.Min(rotateTime + Time.deltaTime * .75f,1);
            transform.rotation = Quaternion.Slerp(rotateFrom, rotateTo, rotateTime);
            if (Quaternion.Angle(rotateTo, transform.rotation) <= .1f)
            {
                rotateTo = transform.rotation;
                rotateFrom = transform.rotation;
                rotateTime = 0;
            }
            UpdateLabelRotations();
        }
    }
}

Answer 1

After a long day of maths, the solution turned out to be quite simple:

Because I knew the spherical coordinates of the labels (eg lat/long), I could figure out the corresponding euler angles I needed, here the relevant code:

InputController.Instance.rotateTo = Quaternion.Euler(
    xzAngle * Mathf.Cos(Mathf.Deg2Rad * yAngle), 
    yAngle, 
    xzAngle * Mathf.Sin(Mathf.Deg2Rad * yAngle)
);

Although posting from several days later, it seems this is not particularily accurate for points outside of multiples of 90 deg longitude