Unity - Project a texture on a mesh using C# (No Shaders)

Question

I'm trying to project a texture on a simple cube meshFilter using only C# but I'm having a bit of a hard time understanding what to do. I almost got it working for the X axis rotation and there is a lot of bad warping for Y/Z. Basically, I update the UVs when the position/rotation of the camera changes, here is my code :

[ExecuteInEditMode]
public class ObjectEditor : MonoBehaviour {

    public GameObject Model;

    public void UpdateTexture(Camera camera) {
        MeshFilter[] mesheFilters = Model.GetComponentsInChildren<MeshFilter>();
        foreach (MeshFilter meshFilter in mesheFilters) {
            int size = meshFilter.sharedMesh.vertices.Length;
            Vector2[] uvs = new Vector2[size];
            for (int i = 0; i < size; i++) {
                uvs[i] = vertexToUVPosition(camera, meshFilter, i);
            }
            meshFilter.sharedMesh.SetUVs(0, uvs);
        }
    }

    private Vector2 vertexToUVPosition(Camera camera, MeshFilter meshFilter, int index) {
        Vector3 vertex = meshFilter.sharedMesh.vertices[index];
        Matrix4x4 VP = camera.projectionMatrix * camera.worldToCameraMatrix;
        Vector4 worldPos = new Vector4(vertex.x, vertex.y, vertex.z, 1f);
        Vector4 clipPos = VP * worldPos;
        clipPos *= 1f / clipPos.w;
        return camera.WorldToScreenPoint(clipPos);
    }
}

Everything regarding the projection happens in vertexToUVPosition. And here is what I have right now (the projected texture is a simple black/white checkerboard) :

Can someone experienced in projections explain to me what I'm doing wrong and maybe provide a sample C# code that works correctly? Thank you.

Answer 1

I found a solution which solves the problem completely but I guess it is not mathematically correct since I don't really know much about matrixes and projections. It is however a good starting point for whoever wants to do something similar without any experience.

Before showing the code, some things you should setup in order to make it easier to debug potential problems :

1. Make sure your texture is in clamp mode, it will be easier to see if the projection works correctly.
2. Position your object at (0,0,0).
3. Position your camera at (0,0,0) and make sure it is in perspective mode.
4. Use another camera in orthographic mode to look at the projected texture and validate it is shown correctly.

The algorithm :

// Calculate the VP matrix based on the Perspective Camera
VP = camera.projectionMatrix * camera.worldToCameraMatrix
foreach vertex in mesh
    // Replace the "w" component by 1.
    worldPosition = new Vector4(vertex.x, vertex.y, vertex.z, 1f);
    clipPosition = VP * worldPosition;
    // Small correction on Y axis (maybe someone can explain why I need this?).
    clipPosition.Scale(new Vector3(1, 0.5f, 1));
    // Use the clipPosition as UV coordinates for that vertex.
    ...

My implementation :

[ExecuteInEditMode]
public class ObjectEditor : MonoBehaviour {

    public GameObject Model;

    public void UpdateTexture(Camera camera) {
        Matrix4x4 vp = camera.projectionMatrix * camera.worldToCameraMatrix;
        MeshFilter[] mesheFilters = Model.GetComponentsInChildren<MeshFilter>();
        foreach (MeshFilter meshFilter in mesheFilters) {
            int size = meshFilter.sharedMesh.vertices.Length;
            Vector2[] uvs = new Vector2[size];
            for (int i = 0; i < size; i++) {
                uvs[i] = vertexToUVPosition(vp, meshFilter, i);
            }
            meshFilter.sharedMesh.SetUVs(0, uvs);
        }
    }

    private Vector2 vertexToUVPosition(Matrix4x4 vp, MeshFilter meshFilter, int index) {
        Vector3 vertex = meshFilter.sharedMesh.vertices[index];
        Vector4 worldPos = new Vector4(vertex.x, vertex.y, vertex.z, 1f);
        Vector4 clipPos = vp * worldPos;
        clipPos.Scale(new Vector3(1, 0.5f, 1));
        return clipPos;
    }
}

The result - Test with rotation on two axis (40,-45,0):

The orthographic camera view :