How to use shader for drawing FBO lwjgl

Question

I am making a 2D game and I want to make a light mask. So far i made a Frame Buffer where i hold my mask but I cannot apply the shader on it. When I don't use the shader the light map is drawn properly (without the scene so for this code i drawns a black screen with one white rectangle) but when i try to blend it with the scene it just draws black screen. This is how i draw the frame buffer and apply the shader.

//draw scene

glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, m_frameBufferObject);//enable frame buffer
            glPushAttrib(GL_VIEWPORT_BIT);
            glViewport(0, 0, 1920, 1080);
            glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);  

            //fill with black
            glColor3f(0.0f, 0.0f, 0.0f);
            glBegin(GL_QUADS);
                glVertex2f(0, 0);
                glVertex2f(1920, 0);
                glVertex2f(1920, 1080);
                glVertex2f(0, 1080);
            glEnd();

            //example light, a rectangle trough which i should see the scene
            glColor3f(1.0f,1.0f,1.0f);
            glBegin(GL_QUADS);
                glVertex2f(100, 100);
                glVertex2f(300, 100);
                glVertex2f(300, 300);
                glVertex2f(100, 300);
            glEnd();

            //Deactivate the framebuffer object
            glPopAttrib();
            glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, 0);

            glUseProgram(shaderProgram3);//this is the shader program
            //Render the framebuffer object to the screen

            glEnable(GL_TEXTURE_2D);
            glBindTexture(GL_TEXTURE_2D, m_frameBufferTexture);
                glBegin(GL_QUADS);
                    glTexCoord2f(0f, 0f);
                    glVertex2f(0, 0);
                    glTexCoord2f(1f, 0f);
                    glVertex2f(1920, 0); 
                    glTexCoord2f(1f, 1f);
                    glVertex2f(1920, 1080);
                    glTexCoord2f(0f, 1f);
                    glVertex2f(0, 1080);
                glEnd();
            glDisable(GL_TEXTURE_2D);
            glUseProgram(0);

    Display.update();
    Display.sync(60);

This is the shader

uniform sampler2D backbuffer;

void main( void ) {    
    vec2 position;
    position.x = gl_FragCoord.x/1920;
    position.y = gl_FragCoord.y/1080;
    vec4 color = texture2D(backbuffer, position);
    gl_FragColor.x -=1-color.x;
    gl_FragColor.y -=1-color.y;
    gl_FragColor.z -=1-color.z;
}

Answer 1

When you do the following in your shader :

gl_FragColor.x -=1-color.x;
gl_FragColor.y -=1-color.y;
gl_FragColor.z -=1-color.z;

You are assuming that when you enter main() gl_FragColor contains the current pixel color of the current framebuffer.

But this is not the case ! gl_FragColor is a color you need to set in the shader and until you set it, its value will be undefined (in your case it seems to be set to black), it will never contain the "current framebuffer color" (check the specs, this applies to all versions of OpenGL / OpenGLES there are no exceptions to this).

To get the current framebuffer color you need to use OpenGL extensions like EXT_shader_framebuffer_fetch for mobile GPUs and sadly this kind of extension does not exist for desktop GPUs (but you can use Image Load Store with OpenGL4).

But here I digress as I do not think you need any of those extensions, what you need instead is simply to enable blending when rendering your mask and configure the blending formulaes correctly to achieve your effect.