How to detect edge of shadow map texture GLSL

Question

I was wondering how I could detect the edges of my shadow map texture in my fragment shader.

I took a look at the following tutorial on youtube: https://www.youtube.com/watch?v=9sEHkT7N7RM But in that guy's case, the solution depends on the camera position. I would prefer not to depend on the camera position.

The problem: If an object moves out of the area that was sampled by the shadow map rendering pass, the object's shadow gets clipped (normal behavior). So, if I could know that the current fragment's position is close to the shadow map border, I could add a fadeout-mechanism of some sort.

Cheers and thanks in advance!

Answer 1

Alright, I actually cannot believe it, but I solved this one myself and it was quite a simple thing to do:

In my vertex shader I calculate the vertex position from the light's point of view and then calculate how close it is to the texture edge:

vShadowCoord = vec4(uShadowModelViewProjectionMatrix * vec4(vertexPos.xyz, 1.0));
float borderFactorX = abs(0.5 - vShadowCoord.x);
float borderFactorY = abs(0.5 - vShadowCoord.y);
vShadowBorderFactor = max(borderFactorX, borderFactorY) * 2;

Now, vShadowBorderFactor stores a value between 0 and 1.

I pass vShadowBorderFactor to the fragment shader and then calculate the amount of darkening (by the casted shadow):

darkening = textureProj(uShadowTexture, vec4(vShadowCoord.xy, vShadowCoord.z - bias, 1.0)) + (vShadowBorderFactor * vShadowBorderFactor);

I then apply the darkening to the scenes overall color vector. Voilà!