Applying a perspective transformation matrix from GIMP into a GLSL shader

Question

So I'm trying to add a rotation and a perspective effect to an image into the vertex shader. The rotation works just fine but I'm unable to make the perspective effect. I'm working in 2D.

The rotation matrix is generated from the code but the perspective matrix is a bunch of hardcoded values I got from GIMP by using the perspective tool.

private final Matrix3 perspectiveTransform = new Matrix3(new float[] {
    0.58302f, -0.29001f, 103.0f,
    -0.00753f, 0.01827f, 203.0f,
    -0.00002f, -0.00115f, 1.0f
});

This perspective matrix was doing the result I want in GIMP using a 500x500 image. I'm then trying to apply this same matrix on texture coordinates. That's why I'm multiplying by 500 before and dividing by 500 after.

attribute vec4 a_position;
attribute vec4 a_color;
attribute vec2 a_texCoord0;

uniform mat4 u_projTrans;
uniform mat3 u_rotation;
uniform mat3 u_perspective;

varying vec4 v_color;
varying vec2 v_texCoords;

void main() {
    v_color = a_color;

    vec3 vec = vec3(a_texCoord0 * 500.0, 1.0);

    vec = vec * u_perspective;

    vec = vec3((vec.xy / vec.z) / 500.0, 0.0);

    vec -= vec3(0.5, 0.5, 0.0);
    vec = vec * u_rotation;
    v_texCoords = vec.xy + vec2(0.5);

    gl_Position = u_projTrans * a_position;
}

For the rotation, I'm offsetting the origin so that it rotates around the center instead of the top left corner.

Pretty much everything I know about GIMP's perspective tool comes from http://www.math.ubc.ca/~cass/graphics/manual/pdf/ch10.ps This was suggesting I would be able to reproduce what GIMP does after reading it, but it turns out I can't. The result shows nothing (no pixel) while removing the perspective part shows the image rotating properly.

As mentioned in the link, I'm dividing by vec.z to convert my homogeneous coordinates back to a 2D point. I'm not using the origin shifting for the perspective transformation as it was mentioned in the link that the top left corner was used as an origin. p.11:

There is one thing to be careful about - the origin of GIMP coordinates is at the upper left, with y increasing downwards.

EDIT:

Thanks to @Rabbid76's answer, it's now showing something! However, it's not transforming my texture like the matrix was transforming my image on GIMP.

My transformation matrix on GIMP was supposed to do something a bit like that:

But instead, it looks like something like that:

This is what I think from what I can see from the actual result:

https://i.sstatic.net/e98UL.jpg (Image used)

(As pointed out, it texture parameter is clamp to edge instead of clamp to border, but that's beside the point)

It looks like it's doing the exact opposite of what I'm looking for. I tried offsetting the origin to the center of the image and to the bottom left before applying the matrix without success. This is a new result but it's still the same problem: How to apply the GIMP perspective matric into a GLSL shader?

EDIT2:

With more testing, I can confirm that it's doing the "opposite". Using this simple downscale transformation matrix:

private final Matrix3 perspectiveTransform = new Matrix3(new float[] {
        0.75f, 0f, 50f,
        0f, 0.75f, 50f,
        0f, 0f, 1.0f
});

The result is an upscaled version of the image:

If I invert the matrix programmatically, it works for the simple scaling matrix! But for the perspective matrix, it shows that:

https://i.sstatic.net/1szLq.jpg

EDIT3:

Thanks to @Rabbid76 again it turned out applying the rotation after the perspective matrix does the rotation before and I end up with a result like this: https://i.sstatic.net/NGrle.jpg

It is almost it! The only problem is that the image is VERY squished. It's just like the perspective matrix was applied multiple times. But if you look carefully, you can see it rotating while in perspective just like I want it. The problem now is how to unsquish it to get a result just like I had in GIMP. (The root problem is still the same, how to take a GIMP matrix and apply it in a shader)

Answer 1

This perspective matrix was doing the result I want in GIMP using a 500x500 image. I'm then trying to apply this same matrix on texture coordinates. That's why I'm multiplying by 500 before and dividing by 500 after.

The matrix

 0.58302 -0.29001 103.0
-0.00753  0.01827 203.0
-0.00002 -0.00115 1.0f

is a 2D perspective transformation matrix. It operates with 2D Homogeneous coordinate.
See 2D affine and perspective transformation matrices

Since the matrix which is displayed in GIMP is the transformation from the perspective to the orthogonal view, the inverse matrix has to be used for the transformation. The inverse matrix can be calculated by calling inv().

The matrix is setup to performs a operation of a Cartesian coordinate in the range [0, 500], to a Homogeneous coordinates in the range [0, 500].

Your assumption is correct, you have to scale the input from the range [0, 1] to [0, 500] and the output from [0, 500] to [0, 1]. But you have to scale the 2D Cartesian coordinates
Further you have to do the rotation after the perspective projection and the Perspective divide.
It may be necessary (dependent on the bitmap and the texture coordinate attributes), that you have to flip the V coordinate of the texture coordinates.

And most important, the transformation has to be done per fragment in the fragment shader. Note, since this transformation is not linear (it is perspective transformation), it is not sufficient to to calculate the texture coordinates on the corner points.

vec2 Project2D( in vec2 uv_coord )
{
    vec2 v_texCoords;

    const float scale = 500.0;

    // flip Y
    //vec2 uv = vec2(uv_coord.x, 1.0 - uv_coord.y);
    vec2 uv = uv_coord.xy;

    // uv_h: 3D homougenus in range [0, 500] 
    vec3 uv_h = vec3(uv * scale, 1.0) * u_perspective;

    // uv_h: perspective devide and downscale [0, 500] -> [0, 1]
    vec3 uv_p = vec3(uv_h.xy / uv_h.z / scale, 1.0);

    // rotate
    uv_p = vec3(uv_p.xy - vec2(0.5), 0.0) * u_rotation + vec3(0.5, 0.5, 0.0);

    return uv_p.xy; 
}

Of course you can do the transformation in the vertex shader too. But then you have to pass the 2d homogeneous coordinate to from the vertex shader to the fragment shader This is similar to set a clip space coordinates to gl_Position. The difference is that you have a 2d homogeneous coordinate and not a 3d. and you have to do the Perspective divide manually in the fragment shader:

Vertex shader:

attribute vec2 a_texCoord0;
varying   vec3 v_texCoords_h;

uniform mat3 u_perspective

vec3 Project2D( in vec2 uv_coord )
{
    vec2 v_texCoords;

    const float scale = 500.0;

    // flip Y
    //vec2 uv = vec2(uv_coord.x, 1.0 - uv_coord.y);
    vec2 uv = uv_coord.xy;

    // uv_h: 3D homougenus in range [0, 500] 
    vec3 uv_h = vec3(uv * scale, 1.0) * u_perspective;

    // downscale
    return vec3(uv_h.xy / scale, uv_h.z);
}

void main()
{
    v_texCoords_h = Project2D( a_texCoord0 );

    .....
}

Fragment shader:

varying vec3 v_texCoords_h;

uniform mat3 u_rotation;

void main()
{
    // perspective divide
    vec2 uv = vertTex.xy / vertTex.z;

    // rotation
    uv = (vec3(uv.xy - vec2(0.5), 0.0) * u_rotation + vec3(0.5, 0.5, 0.0)).xy;

    .....
}

See the preview, where I used the following 2D projection matrix, which is the inverse matrix from that one which is displayed in GIMP:

2.452f,     2.6675f,    -388.0f,
0.0f,       7.7721f,    -138.0f,
0.00001f,   0.00968f,    1.0f

---

Further note, in compare to u_projTrans, u_perspective is initialized in row major order.

Because of that you have to multiply the vector from the left to u_perspective:

vec_h = vec3(vec.xy * 500.0, 1.0) * u_perspective;

But you have to multiply the vector from the right to u_projTrans:

gl_Position = u_projTrans * a_position;

See GLSL Programming/Vector and Matrix Operations
and Data Type (GLSL)

Of course this may change if you transpose the matrix when you set it by glUniformMatrix*