Why is my frag shader casting long shadows horizontally and short shadows vertically?

Question

I have the following fragment shader:

#version 330

layout(location=0) out vec4 frag_colour;

in vec2 texelCoords;

uniform sampler2D uTexture;                 // the color
uniform sampler2D uTextureHeightmap;        // the heightmap
uniform float uSunDistance = -10000000.0;   // really far away vertically
uniform float uSunInclination;              // height from the heightmap plane
uniform float uSunAzimuth;                  // clockwise rotation point
uniform float uQuality;                     // used to determine number of steps and steps size
void main()
{
    vec4 c = texture(uTexture,texelCoords);

    vec2 textureD = textureSize(uTexture,0);
    float d = max(textureD.x,textureD.y);       // use the largest dimension to determine stepsize etc

    // position the sun in the centre of the screen and convert from spherical to cartesian coordinates
    vec3 sunPosition = vec3(textureD.x/2,textureD.y/2,0) + vec3(    uSunDistance*sin(uSunInclination)*cos(uSunAzimuth),
                                                                    uSunDistance*sin(uSunInclination)*sin(uSunAzimuth),
                                                                    uSunDistance*cos(uSunInclination)   );

    float height = texture2D(uTextureHeightmap, texelCoords).r;         // starting height
    vec3 direction = normalize(vec3(texelCoords,height) - sunPosition); // sunlight direction

    float sampleDistance = 0;
    float samples = d*uQuality;
    float stepSize = 1.0 / ((samples/d) * d);

    for(int i = 0; i < samples; i++)
    {
        sampleDistance += stepSize; // increase the sample distance

        vec3 newPoint = vec3(texelCoords,height) + direction * sampleDistance; // get the coord for the next sample point

        float newHeight = texture2D(uTextureHeightmap,newPoint.xy).r;   // get the height of that sample point

        // put it in shadow if we hit something that is higher than our starting point AND is heigher than the ray we're casting
        if(newHeight > height && newHeight > newPoint.z)    
        {
            c *= 0.5;
            break;
        }
    }

    frag_colour = c;
}

The purpose is for it to cast shadows based on a heightmap. Pretty nifty, and the results look good.

However, there's a problem where the shadows appear longer when they are horizontal compared to vertical. If I make the window size different, with a window that is taller than wide, I get the opposite effect. I.e., the shadows are casting longer in the longer dimension.

This tells me that it's to do with the way I'm stepping in the above shader, but I can't tell the problem.

To illustrate, here is the with a uSunAzimuth that results in a horizontally cast shadow:

And here is the exact same code with a uSunAzimuth for a vertical shadow:

It's not very pronounced in these low resolution images, but in larger resolutions the effect gets more exaggerated. Essentially; the shadow when you measure how it casts in all 360 degrees of azimuth clears out an ellipse instead of a circle.

Answer 1

The shadow fragment shader operates on a "snapshot" of the viewport. When your scene is rendered and this "snapshot" is generated, then the vertex positions are transformed by the projection matrix. The projection matrix describes the mapping from 3D points of a scene, to 2D points of the viewport and takes in account the aspect ration of the viewport.
(see Both depth buffer and triangle face orientation are reversed in OpenGL,
and Transform the modelMatrix).

This causes that the high map (uTextureHeightmap) represents a rectangular field of view, dependent on the aspect ratio.
But the texture coordinates, which you use to access the height map describe a quad in the range (0, 0) to (1, 1).
This mismatch must be balanced, by scaling with the aspect ratio.

vec3 direction = ....;

float aspectRatio = textureD.x / textureD.y;
direction.xy *= vec2( 1.0/aspectRatio, 1.0 );