LibGDX Overlapping 2D Shadows

Question

I'm working on shadows for a 2D overhead game. Right now, the shadows are just sprites with the color (0,0,0,0.1) drawn on a layer above the tiles.

The problem: When many entities or trees get clumped together, the shadows overlap, forming unnatural-looking dark areas.

  

I've tried drawing the shadows to a framebuffer and using a simple shader to prevent overlapping, but that lead to other problems, including layering issues. Is it possible to enable a certain blend function for the shadows that prevents "stacking", or a better way to use a shader?

Answer 1

If you don't want to deal with sorting issues, I think you could do this with a shader. But every object will have to be either affected by shadow or not. So tall trees could be marked as not shadow receiving, while the ground, grass, and characters would be shadow receiving.

First make a frame buffer with clear color white. Draw all your shadows on it as pure black.

Then make a shadow mapping shader to draw everything in your world. This relies on you not needing all four channels of the sprite's color, because we need one of those channels to mark each sprite as shadow receiving or not. For example, if you aren't using RGB to tint your sprites, we could use the R channel. Or if you aren't fading them in and out, we could use A. I'll assume the latter here:

Vertex shader:

attribute vec4 a_position;
attribute vec4 a_color;
attribute vec2 a_texCoord0;

varying vec2 v_texCoords;
varying vec2 v_texCoordsShadowmap;
varying vec4 v_color;

uniform mat4 u_projTrans;

void main()
{
    v_texCoords = a_texCoord0;
    v_color = a_color;
    v_color.a = v_color.a * (255.0/254.0); //this is a correction due to color float precision (see SpriteBatch's default shader)
    vec3 screenPosition = u_projTrans * a_position;
    v_texCoordsShadowmap = (screenPosition.xy * 0.5) + 0.5;
    gl_Position = screenPosition;
}

Fragment shader:

#ifdef GL_ES
    precision mediump float;
#endif

varying vec2 v_texCoords;
varying vec2 v_texCoordsShadowmap;
varying vec4 v_color;

uniform sampler2D u_texture;
uniform sampler2D u_textureShadowmap;

void main()
{
    vec4 textureColor = texture2D(u_texture, v_texCoords);
    float shadowColor = texture2D(u_textureShadowmap, v_texCoordsShadowmap).r;
    shadowColor = mix(shadowColor, 1.0, v_color.a);
    textureColor.rgb *= shadowColor * v_color.rgb;
    gl_FragColor = textureColor;
}

These are completely untested and probably have bugs. Make sure you assign the frame buffer's color texture to "u_textureShadowmap". And for all your sprites, set their color's alpha based on how much shadow you want them to have cast on them, which will generally always be 0 or 0.1 (based on the brightness you were using before).