I found this in our internal code as well and I'm trying to understand what is happening.
In the following code: https://github.com/microsoft/DirectX-Graphics-Samples/tree/master/Samples/Desktop/D3D12MeshShaders/src/MeshletRender
They do Transpose(M * V * P) before sending it to the shader. In the shader it's treated as a row-major matrix and they do pos * MVP. Why is this? I have similar code where we multiply the MVP outside in a row-major matrix and then insert it into the shaders row-major matrix, and then we do mul(pos, transpose(mvp)).
We have similar code for PSSL where we do the M * V * P and send it to the shader where we have specified that the matrix is row_major float4x4 but then we don't have to do transpose.
Hopefully someone can help me out here because it's very confusing. Does it have to do with home the memory is handled?
I got confirmed that DX11 is column-major.
On line 32, the combined model-view-projection matrix is computed by multiplying the projection, view, and world matrix together. You will notice that we are post-multiplying the world matrix by the view matrix and the model-view matrix by the projection matrix. If you have done some programming with DirectX in the past, you may have used row-major matrix order in which case you would have swapped the order of multiplications. Since DirectX 10, the default order for matrices in HLSL is column-major so we will stick to this convention in this demo and future DirectX demos.
Using column-major matrices means that we have to post-multiply the vertex position by the model-view-projection matrix to correctly transform the vertex position from object-space to homogeneous clip-space.
From https://www.3dgep.com/introduction-to-directx-11/
Matrix packing order for uniform parameters is set to column-major by default.
Hope this saves someone from going insane.