Is there a way to use Quaternion.FromToRotation to favor using one axis?

Question

I've been struggling with this problem for too many hours and it's time for me to ask for your help.

Situation

I have a moving character I walk around with and align with the surface it walks on. I do this by raycasting down to the next surface it's going to be on and getting the rotation needed to align to the surface with

Quaternion newRotation = Quaternion.FromToRotation(transform.up, foundSurface.normal) * transform.rotation;

The character is a transform called Modelholder that smoothly rotates to the new rotations with

modelHolder.rotation = Quaternion.Slerp(modelHolder.rotation, newRotation, Time.deltaTime * modelRotationSmoothing);

Inside of the modelHolder is the model. I change model.localRotation.y based on mouse jaw movement.

The camera is just a transform that follows the modelholder and has a child transform called Rotator that rotates based on mouse movement too (this time jaw and pitch). The camera is a child of this Rotator transform.

With this, I can walk up walks and ceiling while properly aligning to everything, pretty neat.

Problem

I made a grappling hook which moves the character to grappled surfaces. The realignment at the end of the flight is done in the same way as the walking alignment. This all works fine, except for when you grapple from ground to ceiling (or vice versa). The model seems to do a "barrel roll" to realign to the new surface when I look east or west, but will do a back- or frontflip when looking north or south.

ModelHolder movement: https://streamable.com/qf94k

Model movement: https://streamable.com/4xkl4

The barrel roll is fine, but I want to somehow find a way to rotate so that the player won't be looking in the opposite direction after landing (or jumping from the ceiling while looking north or south, because then realignment to gravity will cause the flip as well) because the flip is terrible disorienting.

Things I've tried:

• I tried having no separation in modelHolder and model. This doesn't solve the problem and only gives me more problems with having a smooth but responsive camera.

• I tried saving the lookdirection before realignment and rotation to the old lookdirection while realigning. This just does a super weird flip and turn thing which is even more disorienting.

• I tried trying to detect differences in comparisons between old and new rotations to see if I can somehow "detect" when it wants to do a flip and when it wants to do a barrel roll so I can counter this. I found only confusion and frustration.

Answer 1

You need to calculate a newRotation that maintains the forward direction as much as possible while having the local up be the normal of the surface.

FromToRotation only guarantees one axis of alignment in the way that you're using it. Instead, you can use cross products and Quaternion.LookRotation to do the calculation you need.

Vector3 newPlayerRight = Vector3.Cross(foundSurface.normal, modelHolder.forward);
Vector3 newPlayerForward = Vector3.Cross(newPlayerRight, foundSurface.normal);

Quaternion newRotation = Quaternion.LookRotation(newPlayerForward, foundSurface.normal);

Then, you can proceed as before:

modelHolder.rotation = Quaternion.Slerp(modelHolder.rotation, newRotation, 
        Time.deltaTime * modelRotationSmoothing);

Although I don't endorse using Slerp/Lerp methods with a tthat isn't guaranteed to ever reach or exceed 1. I would instead recommend using Quaternion.RotateTowards:

float modelRotationSpeed = 180f;
modelHolder.rotation = Quaternion.RotateTowards(modelHolder.rotation, newRotation, 
        Time.deltaTime * modelRotationSpeed);

In order to maintain the forward angle relative to the edge just traversed, you can try a different method:

Quaternion newRotation;

// ..

Vector3 previousSurfaceNormal = modelHolder.up;
Vector3 previousForward = modelHolder.forward;
bool flyingOrFallingToNewSurface;

if (flyingOrFallingToNewSurface)
{
    Vector3 newPlayerRight = Vector3.Cross(foundSurface.normal, modelHolder.forward);
    Vector3 newPlayerForward = Vector3.Cross(newPlayerRight, foundSurface.normal);

    newRotation = Quaternion.LookRotation(newPlayerForward, foundSurface.normal);
} else
{
    // This direction lies in both surfaces. 
    Vector3 edgeTraversed = Vector3.Cross(previousSurfaceNormal, foundSurface.normal);

    // Find the angle from edgeTraversed to previousForward
    float ang = Vector3.SignedAngle(edgeTraversed, previousForward, previousSurfaceNormal);

    // Find newForward in new plane that's the same angle
    Vector3 newPlayerForward = Quaternion.AngleAxis(ang,foundSurface.normal) * edgeTraversed;

    newRotation = Quaternion.LookRotation(newPlayerForward, foundSurface.normal);
}

// ...

float modelRotationSpeed = 180f;
modelHolder.rotation = Quaternion.RotateTowards(modelHolder.rotation, newRotation, 
        Time.deltaTime * modelRotationSpeed);