How to rotate a shape in another 3D software so that the rotation matches with the rotation in Blender?

Question

I import a mesh in Blender, export it with setting axis_forward='-Z', axis_up='Y', import in NVIDIA's FleX, rotate it and store it on disk; I call this mesh X. I also import the mesh in Blender and rotate it the same amount; I call this mesh Y. After I imported X in Blender (setting axis_forward='-Z', axis_up='Y') I realized that X has a different rotation which means that FleX is either using a different global axes (I think it's -Y forward and Z up) or it applies the rotations in a different order or a combination of both. So I am pretty confused now. I do not want to change anything on the FleX side. However, I want to rotate Y in a way that it matches with what FleX exports (X) after rotation. I also tried with Quaternions but I'm still unable to figure out the way FleX is transforming the object. The only thing that I've been able to figure out empirically is that the value for rx rotation is applied the same way for both FleX and Blender. This means that objects X and Y would overlap completely if imported in on scene in Blender.

My goal is to rotate the object in FleX so that object Y's rotation matches exactly with object X when I import X in Blender. For instance, I want FleX rotate the object the same way as Blender rotates it with the Euler rotation vector of [rx, ry, rz]. How can I achieve this?

Note that I am not changing the rotation order in Blender and I'm using the default XYZ rotation.

Here I are some images of the visual differences between FleX and Blender for such transformations:

Euler rotation vector: [0, 89.9, 0] in for object Y in Blender.

After applying the rotation in Blender (object Y):

After applying the same rotation ([0, 89.9, 0]) in FleX and importing the object X in Blender:

Euler rotation vector: [0, 0, 89.9] for object Y in Blender.

After applying the same rotation ([0, 0, 89.9]) in FleX and importing the object X in Blender:

It might look it is easy to guess from the images that FleX swaps the ry and rz when applying the rotation. If I also swap ry and rz when applying the rotations in Blender then both X and Y overlap. However, this only works if all components of the rotation vector are zero except one of them. If the rotation vector is something like [-43.964176, 20.641195, -1.2689421] X and Y do not overlap anymore and discrepancies start to show up as shown below:

The discrepancies are more visible if I put a cloth on X in FleX, import the cloth and Y in Blender and rotate Y by [-43.964176, -1.2689421, 20.641195] (notice that I've swapped ry and rz when applying the rotation in Blender, not in FleX):

Note that if I import X here, the cloth would be perfectly covering it while touching it on the surface:

After doing some hacking for the last example, I noticed that if I apply the rotation vector [-43.964176, 1.2689421, 20.641195] (rx, -rz, ry) the objects overlap almost perfectly:

For another example, I want to apply the rotation [87.68034, 79.94778, 82.697876] in Blender which should ideally give me something like this:

However, FleX is giving me the following (no swapping before passing the rotation vector to FleX):

I was thinking maybe applying the rotation in order of (rx, -rz, ry) in Blender would give me the perfectly overlapping result that I got for the previous example, but instead I got very weird results for another as shown below. Note that I wanted to rotate the object by [87.68034, 79.94778, 82.697876]`:

By manually rotating the object in Blender I could eventually get it as close as possible to Y (exported from FleX). Surprisingly, the rotation vector that overlaps X and Y is way different than [87.68034, 79.94778, 82.697876] in (rx, ry, rz) or [87.68034 -82.697876, 79.94778] in (rx, -rz, ry). It is something around [5.38, -10.1, 88.6] in `(rx, ry, rz) as shown below:

Although you might need more information to figure out exactly what's going out but below I post the code that is used in FleX to compute its rotation matrices. The top one is used when Euler angles are used and the second one is used when Quaternions are input:

// generate a rotation matrix around an axis, from PBRT p74
inline Mat44 RotationMatrix(float angle, const Vec3& axis)
{
    Vec3 a = Normalize(axis);
    float s = sinf(angle);
    float c = cosf(angle);

    float m[4][4];

    m[0][0] = a.x * a.x + (1.0f - a.x * a.x) * c;
    m[0][1] = a.x * a.y * (1.0f - c) + a.z * s;
    m[0][2] = a.x * a.z * (1.0f - c) - a.y * s;
    m[0][3] = 0.0f;

    m[1][0] = a.x * a.y * (1.0f - c) - a.z * s;
    m[1][1] = a.y * a.y + (1.0f - a.y * a.y) * c;
    m[1][2] = a.y * a.z * (1.0f - c) + a.x * s;
    m[1][3] = 0.0f;

    m[2][0] = a.x * a.z * (1.0f - c) + a.y * s;
    m[2][1] = a.y * a.z * (1.0f - c) - a.x * s;
    m[2][2] = a.z * a.z + (1.0f - a.z * a.z) * c;
    m[2][3] = 0.0f;

    m[3][0] = 0.0f;
    m[3][1] = 0.0f;
    m[3][2] = 0.0f;
    m[3][3] = 1.0f;

    return Mat44(&m[0][0]);
}


inline Mat44 RotationMatrix(Quat q)
{
    Matrix33 rotation(q);

    Matrix44 m;
    m.SetAxis(0, rotation.cols[0]);
    m.SetAxis(1, rotation.cols[1]);
    m.SetAxis(2, rotation.cols[2]);
    m.SetTranslation(Point3(0.0f));

    return m;
}

And here's how I apply the Euler rotation vector:

obj->Transform(RotationMatrix(op.rotate.x, Vec3(1.0f, 0.0f, 0.0f)));
obj->Transform(RotationMatrix(op.rotate.y, Vec3(0.0f, 1.0f, 0.0f)));
obj->Transform(RotationMatrix(op.rotate.z, Vec3(0.0f, 0.0f, 1.0f)));

And I think this should be how Transform() is defined:

void Mesh::Transform(const Matrix44& m)
{
    for (uint32_t i=0; i < m_positions.size(); ++i)
    {
        m_positions[i] = m*m_positions[i];
        m_normals[i] = m*m_normals[i];
    }
}

Answer 1

I just had to make the following changes to [manually] change the coordinate system of FleX to the one that Blender uses:

obj->Transform(RotationMatrix(op.rotate.x, Vec3(1.0f, 0.0f, 0.0f)));
obj->Transform(RotationMatrix(op.rotate.y, Vec3(0.0f, 0.0f, -1.0f)))
obj->Transform(RotationMatrix(op.rotate.z, Vec3(0.0f, 1.0f, 0.0f)));

The way I figured this was I generated three rotation vectors [90, 0, 0], [0, 90, 0] and [0, 0, 90] and rotated the shapes in FleX. Then I loaded the shapes in Blender and empirically figured rotation on which axis in Blender matches to the shape imported from FleX. I learned that when I rotate the shape on the Yaxis by 90 degrees in Blender, it matches the 90-degree rotation on the Z axis in FleX. Also, I learned that rotating the object by -90 degrees on the Z axis in Blender matches with the 90-degree rotation on the Y axis in FleX.