C# Opengl Hook Swapbuffer to add interpolate frame

Question

I'm trying to run a game in 60Fps instead of 30. The game is lock to 30 fps, so the only way I can reach 60 fps is with frame interpolation.

For the interpolation part, I'm going to use OpenCV, I found this pretty good article.

This game use OpenGL. After some research, I found that the best way to grab screen was to hook SwapBuffer function. So I choose to hook the game to grab screens and send them to an other app, which will stream the game in real time and by the way will add interpolate frames to reach 60 FPS (If you have a better idea, I'm fully open !)

I started to write my new dll. I'm coding in C#, and I chose EasyHook to inject my DLL.

Hooking is working well, pretty cool right now =D.

However, I'm now stuck because I have absolutely no idea of how to grab screen from the game.

I tried to use OpenGL.Net and SharpGL, but I can't figure out how I'm suppose to grab actual OpenGL context to use glReadPixel!

Here is my actual code:

using System;
using System.Runtime.InteropServices;


namespace Hook
{
    public class ServerInterface : MarshalByRefObject
    {
        public void IsInstalled(int clientPID)
        {
            Console.WriteLine("This programm has injected dll into process {0}.\r\n", clientPID);
        }

        public void ReportMessage(int clientPID, string message)
        {
            Console.WriteLine(message);
        }

        public void ReportException(Exception e)
        {
            Console.WriteLine("The target process has reported an error:\r\n" + e.ToString());
        }

        public void Ping()
        {
            Console.WriteLine("Ping !");
        }
    }


    //Point d'entrée
    public class InjectionEntryPoint : EasyHook.IEntryPoint
    {
        //Serveur :
        ServerInterface server = null;

        public InjectionEntryPoint(EasyHook.RemoteHooking.IContext context, string channelName)
        {
            //Constructeur
            //Objectif : vérifier si la communication entre les deux programmes peut etre établit
            server = EasyHook.RemoteHooking.IpcConnectClient<ServerInterface>(channelName);
            server.Ping();

        }

        public void Run(EasyHook.RemoteHooking.IContext context, string channelName)
        {

            try
            {
                var SwapBuffersHook = EasyHook.LocalHook.Create(EasyHook.LocalHook.GetProcAddress("opengl32.dll", "wglSwapBuffers"), new SwapBuffers_Delegate(SwapBuffers_Hook), this);
                SwapBuffersHook.ThreadACL.SetExclusiveACL(new Int32[] { 0 });
                server.ReportMessage(EasyHook.RemoteHooking.GetCurrentProcessId(), "SwapBuffers Hooked");
            }
            catch (Exception ExtInfo)
            {
                server.ReportException(ExtInfo);
                return;
            }

            server.IsInstalled(EasyHook.RemoteHooking.GetCurrentProcessId());

            EasyHook.RemoteHooking.WakeUpProcess();

            //Waiting years
            while( true)
            {
                System.Threading.Thread.Sleep(10000);
            }


            // Finalise cleanup of hooks
            EasyHook.LocalHook.Release();

        }


        //Deleguate
        [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
        delegate IntPtr SwapBuffers_Delegate(IntPtr hdc);

        //Original
        [DllImport("opengl32.dll", CharSet = CharSet.Unicode, SetLastError = true, CallingConvention = CallingConvention.StdCall)]
        static extern IntPtr wglSwapBuffers(IntPtr hdc);

        //Hook function
        public IntPtr SwapBuffers_Hook(IntPtr hdc)
        {

            server.ReportMessage(EasyHook.RemoteHooking.GetCurrentProcessId(),  "I'm in SwapBuffers =DDDD");

            //Here I need to grab frames

            //Return
            return wglSwapBuffers(hdc);
        }
    }
}

Answer 1

I doubt you can increase the FPS of the game. By the way, there are some possibilities.

What are you doing is the right way. You just miss the GL code to create the intermediate frame to be displayed (and I suggest to create it in-process, using the current GL context).

The problem is that the video frames are displayed when the SwapBuffers command is executed. This is the one you hook, but ask to yourself and to the program: why the SwapBuffers is called 30 times per second?

If the answer is "Because the application suspend execution using timers", you cannot solve it easily: your hook must work in conjunction with the application, hence you need to understand how the application sleeps.

If the answe is "Because the application is taking advantage of WGL_EXT_swap_control", than you have one possibility:

• Set wglSwapIntervalEXT(1) (because the application has called wglSwapIntervalEXT(2) to get a 30 FPS synced with V-Sync. Probably this is necessary only once
• In your hook, grab two frames and then interpolate them at every other frame (hence your question). At each hook execution, you need to render the interpolated frame, then swap; then render the next original frame, then swap.

How to create interpolated frame? If you are doing via CPU, use glReadPixels with the back buffer (GL_BACK) as read framebuffer (using glDrawBuffers). This will download the content of the framebuffer bound. Then use blending equation to get interpolated pixels.