How to render a texture to an Android GLSurfaceView

Question

I have a standard GLSurfaceView class:

public class TestSurfaceView extends GLSurfaceView {
    public MainRenderer mRenderer;

    public GStreamerSurfaceView(Context context) {
        super(context);

        setEGLContextClientVersion(2);
        mRenderer = new MainRenderer(context);
        setRenderer(mRenderer);
        setRenderMode(GLSurfaceView.RENDERMODE_CONTINUOUSLY);
    }
}

I have a Renderer class that implements GLSurfaceView.Renderer:

public class MainRenderer implements GLSurfaceView.Renderer {
    private int[] hTex;
    private SurfaceTexture mSTexture;
    private Context context;

    MainRenderer(Context c) {
        context = c;
    }

    public void onSurfaceCreated(GL10 unused, EGLConfig config) {
    }

    public void onDrawFrame(GL10 unused) {
    }

    public void onSurfaceChanged(GL10 unused, int width, int height) {
    }

    public void onSurfaceCreated(GL10 arg0, javax.microedition.khronos.egl.EGLConfig arg1) {
    }
}

In a separate JNI thread, I have some video data (YUV format) uploaded to an OpenGLES texture. I receive in Java the notification that a new texture is available and I have the corresponding texture id.

How can I display the content of this texture in the Renderer class with minimum performance impact?

Answer 1

For cases where the frames are coming from Camera or MediaCodec there are some very efficient solutions. It sounds like you're generating or decoding the video in software, though, which puts a mild spin on things (though you do have a SurfaceTexture declared in your code, which is odd). The trick is the "OpenGL ES texture" part, because the texture is associated with the EGL context, and the EGL context can only be active in one thread at a time.

Because you're using GLSurfaceView, rather than plain SurfaceView, you don't have control over the EGL context in the GLSurfaceView's renderer thread. The easiest way to work around this is to jump through some hoops to create a second EGL context that is shared with the first. Once you've done that, the texture created in the separate JNI thread will be available to the GLSurfaceView renderer thread.

You can find an example of this in Grafika's "show + capture camera" activity. If you look at the onDrawFrame() method in CameraCaptureActivity.java you can see it calling updateSharedContext(), which passes a message to the thread running TextureMovieEncoder to cause it to run handleUpdateSharedContext(), which (re-)creates the surface used to feed a video encoder.

If you use plain SurfaceView instead, and do your own EGL and thread management, the code will be less twisty. You can create both contexts at once, then just pass one to the thread that's producing images. You could also create a single context and use eglMakeCurrent() to shift it between threads, but that could be expensive on some platforms.

Update: the Grafika "show + capture camera" implementation has a race condition; see this bug report for details on the problem and solution. You have to perform some extra steps when creating a texture in one thread and using it in another. It's usually better to do everything in one context on one thread. The other activities in Grafika use a plain SurfaceView and do their own EGL context and thread management.