How to make smoother borders using Fragment shader in OpenGL?

Question

I have been trying to draw border of an Image with transparent background using OpenGL in Android. I am using Fragment Shader & Vertex Shader. (From the GPUImage Library)

Below I have added Fig. A & Fig B.

Fig A.

Fig B.

I have achieved Fig A. With the customised Fragment Shader. But Unable to make the border smoother as in Fig B. I am attaching the Shader code that I have used (to achieve rough border). Can someone here help me on how to make the border smoother?

Here is my Vertex Shader :

    attribute vec4 position;
    attribute vec4 inputTextureCoordinate;        
    varying vec2 textureCoordinate;
    
    void main()
    {
        gl_Position = position;
        textureCoordinate = inputTextureCoordinate.xy;
    }

Here is my Fragment Shader :

I have calculated 8 pixels around the current pixel. If any one pixel of those 8 is opaque(having alpha greater than 0.4), It it drawn as a border color.

                precision mediump float;
                uniform sampler2D inputImageTexture;
                varying vec2 textureCoordinate;
                uniform lowp float thickness;
                uniform lowp vec4 color;

                void main() {
                    float x = textureCoordinate.x;
                    float y = textureCoordinate.y;
                    vec4 current = texture2D(inputImageTexture, vec2(x,y));

                    if ( current.a != 1.0 ) {
                        float offset = thickness * 0.5;
                        
                        vec4 top = texture2D(inputImageTexture, vec2(x, y - offset));
                        vec4 topRight = texture2D(inputImageTexture, vec2(x + offset,y - offset));
                        vec4 topLeft = texture2D(inputImageTexture, vec2(x - offset, y - offset));
                        vec4 right = texture2D(inputImageTexture, vec2(x + offset, y ));
                        vec4 bottom = texture2D(inputImageTexture, vec2(x , y + offset));
                        vec4 bottomLeft  = texture2D(inputImageTexture, vec2(x - offset, y + offset));
                        vec4 bottomRight = texture2D(inputImageTexture, vec2(x + offset, y + offset));
                        vec4 left = texture2D(inputImageTexture, vec2(x - offset, y ));
                        
                        if ( top.a > 0.4 || bottom.a > 0.4 || left.a > 0.4 || right.a > 0.4 || topLeft.a > 0.4 || topRight.a > 0.4 || bottomLeft.a > 0.4 || bottomRight.a > 0.4 ) {
                             if (current.a != 0.0) {
                                 current = mix(color , current , current.a);
                             } else {
                                 current = color;
                             }
                        }
                    }
                    
                    gl_FragColor = current;
                }

Answer 1

You were almost on the right track.

The main algorithm is:

• Blur the image.
• Use pixels with opacity above a certain threshold as outline.

The main problem is the blur step. It needs to be a large and smooth blur to get the smooth outline you want. For blurring, we can use convolution filter Kernel. And to achieve a large blur, we should use a large kernel. And I suggest using the Gaussian Blur distribution, as it is very well known and used.

---

The Overview of he Algorithm is:

For each fragment, we sample many locations around it. The samples are made in an N by N grid. We average them together using weights that follow a 2D Gaussian Distribution. This results in a blurred image.

With the blurred image, we paint the fragments that have alpha greater than a threshold with our outline color. And, of course, any opaque pixels in the original image should also appear in the result.

---

On a sidenote, your solution is almost a blur with a 3 x 3 kernel (you sample locations around the fragment in a 3 by 3 grid). However, a 3 x 3 kernel won't give you the amount of blur you need. You need more samples (e.g. 11 x 11). Also, the weights closer to the center should have a greater impact on the result. Thus, uniform weights won't work very well.

---

Oh, and one more important thing:

One single shader to acomplish this is NOT the fastest way to implement this. Usually, this would be acomplished with 2 separate renders. The first one would render the image as usual and the second render would blur and add the outline. I assumed that you want to do this with 1 single render.

The following is a vertex and fragment shader that accomplish this:

Vertex Shader

varying vec2 vecUV;
varying vec3 vecPos;
varying vec3 vecNormal;

void main() {
    vecUV = uv * 3.0 - 1.0;
    vecPos = (modelViewMatrix * vec4(position, 1.0)).xyz;
    vecNormal = (modelViewMatrix * vec4(normal, 0.0)).xyz;

    gl_Position = projectionMatrix * vec4(vecPos, 1.0);
}

Fragment Shader

precision highp float;

varying vec2 vecUV;
varying vec3 vecPos;
varying vec3 vecNormal;

uniform sampler2D inputImageTexture;


float normalProbabilityDensityFunction(in float x, in float sigma)
{
    return 0.39894*exp(-0.5*x*x/(sigma*sigma))/sigma;
}

vec4 gaussianBlur()
{
    // The gaussian operator size
    // The higher this number, the better quality the outline will be
    // But this number is expensive! O(n2)
    const int matrixSize = 11;
    
    // How far apart (in UV coordinates) are each cell in the Gaussian Blur
    // Increase this for larger outlines!
    vec2 offset = vec2(0.005, 0.005);
    
    const int kernelSize = (matrixSize-1)/2;
    float kernel[matrixSize];
    
    // Create the 1-D kernel using a sigma
    float sigma = 7.0;
    for (int j = 0; j <= kernelSize; ++j)
    {
        kernel[kernelSize+j] = kernel[kernelSize-j] = normalProbabilityDensityFunction(float(j), sigma);
    }
    
    // Generate the normalization factor
    float normalizationFactor = 0.0;
    for (int j = 0; j < matrixSize; ++j)
    {
        normalizationFactor += kernel[j];
    }
    normalizationFactor = normalizationFactor * normalizationFactor;
    
    // Apply the kernel to the fragment
    vec4 outputColor = vec4(0.0);
    for (int i=-kernelSize; i <= kernelSize; ++i)
    {
        for (int j=-kernelSize; j <= kernelSize; ++j)
        {
            float kernelValue = kernel[kernelSize+j]*kernel[kernelSize+i];
            vec2 sampleLocation = vecUV.xy + vec2(float(i)*offset.x,float(j)*offset.y);
            vec4 sample = texture2D(inputImageTexture, sampleLocation);
            outputColor += kernelValue * sample;
        }
    }
    
    // Divide by the normalization factor, so the weights sum to 1
    outputColor = outputColor/(normalizationFactor*normalizationFactor);
    
    return outputColor;
}


void main()
{
    // After blurring, what alpha threshold should we define as outline?
    float alphaTreshold = 0.3;
    
    // How smooth the edges of the outline it should have?
    float outlineSmoothness = 0.1;
    
    // The outline color
    vec4 outlineColor = vec4(1.0, 1.0, 1.0, 1.0);
    
    // Sample the original image and generate a blurred version using a gaussian blur
    vec4 originalImage = texture2D(inputImageTexture, vecUV);
    vec4 blurredImage = gaussianBlur();
    
    
    float alpha = smoothstep(alphaTreshold - outlineSmoothness, alphaTreshold + outlineSmoothness, blurredImage.a);
    vec4 outlineFragmentColor = mix(vec4(0.0), outlineColor, alpha);
    
    gl_FragColor = mix(outlineFragmentColor, originalImage, originalImage.a);
}

This is the result I got:

And for the same image as yours, with matrixSize = 33, alphaTreshold = 0.05

And to try to get crispier results we can tweak the parameters. Here is an example with matrixSize = 111, alphaTreshold = 0.05, offset = vec2(0.002, 0.002), alphaTreshold = 0.01, outlineSmoothness = 0.00. Note that increasing matrixSize will heavily impact performance, which is a limitation of rendering this outline with only one shader pass.

I tested the shader on this site. Hopefully you will be able to adapt it to your solution.

Regarding references, I have used quite a lot of this shadertoy example as basis for the code I wrote for this answer.