Search code examples
opengllibgdxglslshaderblur

Fade texture inner borders to transparent in LibGDX using openGL shaders (glsl)

I'm currently working on a tile game in LibGDX and I'm trying to get a "fog of war" effect by obscuring unexplored tiles. The result I get from this is a dynamically generated black texture of the size of the screen that only covers unexplored tiles leaving the rest of the background visible. This is an example of the fog texture rendered on top of a white background:

Fog texture rendered on top of a white background

What I'm now trying to achieve is to dynamically fade the inner borders of this texture to make it look more like a fog that slowly thickens instead of just a bunch of black boxes put together on top of the background.

Googling the problem I found out I could use shaders to do this, so I tried to learn some glsl (I'm at the very start with shaders) and I came up with this shader:

VertexShader:

//attributes passed from openGL
attribute vec3 a_position;
attribute vec2 a_texCoord0;

//variables visible from java
uniform mat4 u_projTrans;

//variables shared between fragment and vertex shader
varying vec2 v_texCoord0;

void main() {

    v_texCoord0 = a_texCoord0;
    gl_Position = u_projTrans * vec4(a_position, 1f);
}

FragmentShader:

//variables shared between fragment and vertex shader
varying vec2 v_texCoord0;

//variables visible from java
uniform sampler2D u_texture;
uniform vec2 u_textureSize;
uniform int u_length;

void main() {

    vec4 texColor = texture2D(u_texture, v_texCoord0);
    vec2 step = 1.0 / u_textureSize;

    if(texColor.a > 0) {

        int maxNearPixels = (u_length * 2 + 1) * (u_length * 2 + 1) - 1;
        for(int i = 0; i <= u_length; i++) {

            for(float j = 0; j <= u_length; j++) {

                if(i != 0 || j != 0) {

                    texColor.a -= (1 - texture2D(u_texture, v_texCoord0 + vec2(step.x * float(i), step.y * float(j))).a) / float(maxNearPixels);
                    texColor.a -= (1 - texture2D(u_texture, v_texCoord0 + vec2(-step.x * float(i), step.y * float(j))).a) / float(maxNearPixels);
                    texColor.a -= (1 - texture2D(u_texture, v_texCoord0 + vec2(step.x * float(i), -step.y * float(j))).a) / float(maxNearPixels);
                    texColor.a -= (1 - texture2D(u_texture, v_texCoord0 + vec2(-step.x * float(i), -step.y * float(j))).a) / float(maxNearPixels);
                }
            }
        }
    }

    gl_FragColor = texColor;
}

This is the result I got setting a length of 20:

Faded fog

So the shader I wrote kinda works, but has terrible performance cause it's O(n^2) where n is the length of the fade in pixels (so it can be very high, like 60 or even 80). It also has some problems, like that the edges are still a bit too sharp (I'd like a smother transition) and some of the angles of the border are less faded than others (I'd like to have a fade uniform everywhere).

I'm a little bit lost at this point: is there anything I can do to make it better and faster? Like I said I'm new to shaders, so: is it even the right way to use shaders?

Solution

  • As others mentioned in the comments, instead of blurring in the screen-space, you should filter in the tile-space while potentially exploiting the GPU bilinear filtering. Let's go through it with images.

    First define a texture such that each pixel corresponds to a single tile, black/white depending on the fog at that tile. Here's such a texture blown up:

    tile map

    After applying the screen-to-tiles coordinate transformation and sampling that texture with GL_NEAREST interpolation we get the blocky result similar to what you have:

    float t = texture2D(u_tiles, M*uv).r;
gl_FragColor = vec4(t,t,t,1.0);

    nearest

    If instead of GL_NEAREST we switch to GL_LINEAR, we get a somewhat better result:

    linear

    This still looks a little blocky. To improve on that we can apply a smoothstep:

    float t = texture2D(u_tiles, M*uv).r;
t = smoothstep(0.0, 1.0, t);
gl_FragColor = vec4(t,t,t,1.0);

    smoothstep

    Or here is a version with a linear shade-mapping function:

    float t = texture2D(u_tiles, M*uv).r;
t = clamp((t-0.5)*1.5 + 0.5, 0.0, 1.0);
gl_FragColor = vec4(t,t,t,1.0);

    clamp

    Note: these images were generated within a gamma-correct pipeline (i.e. sRGB framebuffer enabled). This is one of those few scenarios, however, where ignoring gamma may give better results, so you're welcome to experiment.