Vertex position relative to normal

Question

In a surface shader, given the world's up axis (and the others too), a world space position and a normal in world space, how can we rotate the worldspace position into the space of the normal?

That is, given a up vector and a non-orthogonal target-up vector, how can we transform the position by rotating its up vector?

I need this so I can get the vertex position only affected by the object's rotation matrix, which I don't have access to.

Here's a graphical visualization of what I want to do:

• Up is the world up vector
• Target is the world space normal
• Pos is arbitrary

The diagram is bidimensional, but I need to solve this for a 3D space.

Answer 1

Looks like you're trying to rotate pos by the same rotation that would transform up to new_up.

Using the rotation matrix found here, we can rotate pos using the following code. This will work either in the surface function or a supplementary vertex function, depending on your application:

// Our 3 vectors
float3 pos;
float3 new_up;
float3 up = float3(0,1,0);

// Build the rotation matrix using notation from the link above
float3 v = cross(up, new_up);
float s = length(v);  // Sine of the angle
float c = dot(up, new_up); // Cosine of the angle
float3x3 VX = float3x3(
    0, -1 * v.z, v.y,
    v.z, 0, -1 * v.x,
    -1 * v.y, v.x, 0
); // This is the skew-symmetric cross-product matrix of v
float3x3 I = float3x3(
    1, 0, 0,
    0, 1, 0,
    0, 0, 1
); // The identity matrix
float3x3 R = I + VX + mul(VX, VX) * (1 - c)/pow(s,2) // The rotation matrix! YAY!

// Finally we rotate
float3 new_pos = mul(R, pos);

This is assuming that new_up is normalized.

If the "target up normal" is a constant, the calculation of R could (and should) only happen once per frame. I'd recommend doing it on the CPU side and passing it into the shader as a variable. Calculating it for every vertex/fragment is costly, consider what it is you actually need.

If your pos is a vector-4, just do the above with the first three elements, the fourth element can remain unchanged (it doesn't really mean anything in this context anyway).

I'm away from a machine where I can run shader code, so if I made any syntactical mistakes in the above, please forgive me.