Cell Shading LibGdx

Im working on my 3D Game with LibGdx.
After looking some other Threads and posts with some realy good tutorials I get the first shader working. My Problem now is to get a Cel/Outline/Toon Shader working. Therefore I found also a tutorial and a project but they havnt worked.

After reading some posts how to solve this shading Problem (with rendering the Object twice,...) I tried this method but got some sideeffects.

Actually I got as result a darfred rendering Scene. My Question is now, If my Models just need some other Material or why I get these results.

Solution

I wrote a cel shader based on the KBAL tutorial that results in renderings like the one above. I've been meaning to write up something on it since the library has changed a lot since then. It seems like you got stuck on the depth shader, which is one of the parts from the original tutorial that needed the most updating.

Besides compatibility updates, I removed one render pass by modifying the uber shader that comes with LibGDX to perform the discretization in the KBAL tutorial's toonify() function during the initial rendering of geometry rather than in a post pass. Aside from that it follows the same pattern.

The code below is a bare bones implementation of my cel shader code. The class is derived extends AbstractScreen which implements some base functionality for LibGDX's Screen interface. Read more about Screen's here and see the CelTutorialScreen source within a full project context here.

package com.hh.ghoststory.screen;

import com.badlogic.gdx.Gdx;
import com.badlogic.gdx.assets.AssetManager;
import com.badlogic.gdx.graphics.GL20;
import com.badlogic.gdx.graphics.PerspectiveCamera;
import com.badlogic.gdx.graphics.Pixmap;
import com.badlogic.gdx.graphics.g2d.SpriteBatch;
import com.badlogic.gdx.graphics.g2d.TextureRegion;
import com.badlogic.gdx.graphics.g3d.Environment;
import com.badlogic.gdx.graphics.g3d.Model;
import com.badlogic.gdx.graphics.g3d.ModelBatch;
import com.badlogic.gdx.graphics.g3d.ModelInstance;
import com.badlogic.gdx.graphics.g3d.environment.DirectionalLight;
import com.badlogic.gdx.graphics.glutils.FrameBuffer;
import com.badlogic.gdx.graphics.glutils.ShaderProgram;
import com.badlogic.gdx.math.Matrix4;
import com.badlogic.gdx.utils.Array;
import com.hh.ghoststory.GhostStory;
import com.hh.ghoststory.render.shaders.CelDepthShaderProvider;
import com.hh.ghoststory.render.shaders.CelLineShaderProgram;

public class CelTutorialScreen extends AbstractScreen {
    private PerspectiveCamera camera = new PerspectiveCamera(67, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());

    private AssetManager assetManager = new AssetManager();
    private Array<ModelInstance> instances = new Array<ModelInstance>();

    private FrameBuffer fbo;
    private TextureRegion textureRegion;
    private ShaderProgram lineShader = new CelLineShaderProgram();

    private SpriteBatch spriteBatch = new SpriteBatch();
    private ModelBatch modelBatch = new ModelBatch(Gdx.files.classpath("com/badlogic/gdx/graphics/g3d/shaders/default.vertex.glsl").readString(), Gdx.files.internal("shaders/cel.main.fragment.glsl").readString());
    private ModelBatch depthBatch = new ModelBatch(new CelDepthShaderProvider());
    private Environment environment = new Environment();

    public CelTutorialScreen(GhostStory game) {
        super(game);

        Gdx.gl.glClearColor(1.0f, 0.0f, 1.0f, 0.0f);
        Gdx.gl.glClear(GL20.GL_DEPTH_BUFFER_BIT | GL20.GL_COLOR_BUFFER_BIT);

        // setup camera
        camera.position.set(5, 5, 5);
        camera.lookAt(0, 0, 0);
        camera.near = 1;
        camera.far = 1000;
        camera.update();

        // add a light
        environment.add(new DirectionalLight().set(0.8f, 0.8f, 1.8f, -1f, -0.8f, 0.2f));

        // load our model
        assetManager.load("models/spider.g3dj", Model.class);
        loading = true;
    }
    @Override
    public void render(float delta) {
        if (loading && assetManager.update())
            doneLoading();

        camera.update();
        Gdx.gl.glClearColor(1.0f, 0.0f, 0.0f, 0.0f);
        Gdx.gl.glClear(GL20.GL_DEPTH_BUFFER_BIT | GL20.GL_COLOR_BUFFER_BIT);

        // render depth map to fbo
        captureDepth();
        // draw the scene
        renderScene();
        // put fbo texture in a TextureRegion and flip it
        prepTextureRegion();
        // draw the cel outlines
        drawOutlines();
    }
    /*
     * Draws the cel outlines using the CelLineShaderProgram
     */
    protected void drawOutlines() {
        spriteBatch.setShader(lineShader);
        lineShader.setUniformf("u_size", Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
        spriteBatch.begin();
        spriteBatch.draw(textureRegion, 0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
        spriteBatch.end();
        spriteBatch.setShader(null);
    }
    /*
     * Stores fbo texture in a TextureRegion and flips it vertically.
     */
    protected void prepTextureRegion() {
        textureRegion = new TextureRegion(fbo.getColorBufferTexture());
        textureRegion.flip(false, true);
    }
    /*
     * Draws the depth pass to an fbo, using a ModelBatch created with CelDepthShaderProvider()
     */
    protected void captureDepth() {
        fbo.begin();
        Gdx.gl.glClearColor(1.0f, 1.0f, 1.0f, 0.0f);
        Gdx.gl.glClear(GL20.GL_DEPTH_BUFFER_BIT | GL20.GL_COLOR_BUFFER_BIT);
        depthBatch.begin(camera);
        depthBatch.render(instances);
        depthBatch.end();
        fbo.end();
    }
    /*
     * Renders the scene.
     */
    protected void renderScene() {
        Gdx.gl.glClear(GL20.GL_DEPTH_BUFFER_BIT | GL20.GL_COLOR_BUFFER_BIT);
        modelBatch.begin(camera);
        modelBatch.render(instances, environment);
        modelBatch.end();
    }
    @Override
    protected void doneLoading() {
        loading = false;
        instances.add(new ModelInstance(assetManager.get("models/spider.g3dj", Model.class)));
    }
    /*
     * Set camera width and height, SpriteBatch projection matrix, and reinit the FBOs
     */
    @Override
    public void resize(int width, int height) {
        camera.position.set(camera.position);
        camera.viewportWidth = width;
        camera.viewportHeight = height;
        camera.update();

        if (fbo != null) fbo.dispose();
        fbo = new FrameBuffer(Pixmap.Format.RGBA8888, Gdx.graphics.getWidth(), Gdx.graphics.getHeight(), true);

        spriteBatch.setProjectionMatrix(new Matrix4().setToOrtho2D(0, 0, width, height));
    }
    @Override
    public void dispose() {
        assetManager.dispose();
        modelBatch.dispose();
        depthBatch.dispose();
        spriteBatch.dispose();
        fbo.dispose();
        lineShader.dispose();
    }
}

The render performs 3 passes to create the end product.

The first is contained in the captureDepth() function.

    protected void captureDepth() {
        fbo.begin();
        Gdx.gl.glClearColor(1.0f, 1.0f, 1.0f, 0.0f);
        Gdx.gl.glClear(GL20.GL_DEPTH_BUFFER_BIT | GL20.GL_COLOR_BUFFER_BIT);
        depthBatch.begin(camera);
        depthBatch.render(instances);
        depthBatch.end();
        fbo.end();
    }

A framebuffer is started, glClear is called and then the depthBatch ModelBatch() renders the model instances (only one in this case) before the framebuffer is ended.

The depthBatch is a ModelBatch that uses a CelDepthShaderProvider, which provides a CelDepthShader. CellDepthShaderProvider is a small class that extends BaseShaderProvider and overrides createShader to return an instance of CelDepthShader, which registers and sets u_near and u_far uniforms as well as sets up the use of the cel depth vertex and fragment GLSL shaders.

I'm guessing the GLSL files are where you're running into issues. The vertex shader I linked to is the same as the KBAL vertex shader with the exception of line 125, which removed some artifacts on the cel edges:

v_depth = (pos.z + u_near) / (u_far - u_near);

The fragment shader is very similar to the one in the KBAL tutorial, but is actually copied from LibGDX's built in depth fragment shader. It's quite possible that the current LigGDX DepthShader could be used instead of my CelDepthShader, but I haven't had time to look into this.

After the first pass, the packed depth map has been captured by the FBO. The second pass is ready to be run and will draw the scene with LibGDXs' default vertex shader and a slightly modified version of its fragment shader.

The changes from the default fragment shader are in lines 140-150 where the specular value is discretized before being added to gl_FragColor:

if (specIntensity > 0.6)
    specFactor = 1.0;
else if (specIntensity > 0.3)
    specFactor = 0.5;
else
    specFactor = 0.1;

specular *= specFactor;

And 173-182 where the overall gl_FragColor is discretized:

float intensity = max(gl_FragColor.r, max(gl_FragColor.g, gl_FragColor.b));
float factor;
if (intensity > 0.8)
    factor = 1.0;
else if (intensity > 0.5)
    factor = 0.8;
else if (intensity > 0.25)
    factor = 0.3;
else
    factor = 0.1;

And that's it for the main cel pass.

Next in render() the prepTextureRegion() function is called. This just puts the depth texture captured to our fbo into a texture region and flips it vertically before using it to draw the cel outlines in the final pass.

The final pass is performed in drawOutlines() and makes use of a SpriteBatch since we're drawing a 2d texture instead of geometry. The call to spriteBatch.setShader(lineshader) sets the SpriteBatch to use an instance of CelLineShaderProgram, another class that extends ShaderProgram. It sets a u_size uniform and uses cel.line.vertex.glsl and cel.line.fragment.glsl.

This shader program runs the Laplace filter. The vertex shader is copied from the KBAL edge shader and updated to work with newer versions of LibGDX, it passes the sampled coordinate of the depth map as well as its top, bottom, left and right neighboring texels to the fragment shader as varyings.

The fragment shader uses an updated method of unpacking the depth values based on code from the getShadowness() function here as recommended by Xoppa.

There are some improvements to this process that could be made. For one, I haven't implemented the super sampling in the original tutorial.

Also, it's not really noticeable in this still image, but once you have a controllable camera in the scene, or geometry moving around, you'll notice the per pixel lighting looks a little weird, especially with limited polygons in your geometry. There is a per-pixel lighting fragment shader in the LibGDX shadow system tests that could be used as a base to implement this with cel shading. The shadow systems might even be a good base to create a multi-pass rendering system for cel shading. And there is undoubtedly code that could be removed from the modified base LibGDX shaders I've used, as well as other optimizations and cleanup.

Hope this helps you or anyone else looking for info on multipass cel shading.