C# Anti-Debug method OutputDebugString doesn't work right

Question

So recently I've been reading up on anti-debug mechanisms and a popular method I've come across to check if the current process is being debugged is OutputDebugString. I've written this piece of code but it doesn't exactly work as intended can someone shed some light on why or what I'm doing wrong?

private static bool stub_OutputDebugString()
{
    uint ErrCode = 0x12A6;
    Native.SetLastError(ErrCode);
    Native.OutputDebugString("System.Core\n");
    if (Marshal.GetLastWin32Error() == (int)ErrCode)
    {
        //Debugger Detected
        return true;
    }
    else
    {
        //No Debugger Detected
        return false;
    }
}

My P/Invoke signatures

[DllImport("kernel32.dll", SetLastError = true)]
public static extern void SetLastError(uint dwErrCode);

[DllImport("kernel32.dll", SetLastError = true)]
public static extern void OutputDebugString(string lpOutputString);

Note I readup on how GetLastError shouldn't be called from the native environment as the value can be changed so I'm using Marshal.GetLastWin32Error()

The code should work but the last error doesn't change when i try to debug the application with windbg or any other debugger.

Answer 1

Your reasoning about GetLastError() (it shouldn't be used because last error may be overwritten by .NET Framework because of any other internal/invisible/background operation) also applies to SetLastError(). Simply you can't reliably call it and expect its value is preserved intact until you call Marshal.GetLastWin32Error().

We may discuss how to circumvent this issue however...

...that technique has (somehow) sense for native code because OutputDebugString() is widely used and disassembling compiled code is harder (then you protect yourself with some obfuscated code). However managed code is easy to decompile and it's still easy to read and understand, such simple obfuscation will not help and you should simply go with System.Diagnostics.Debugger.IsAttached property.

---

If you really really want to detect a debugger (even if it's very easy to see what you're doing then it won't offer much protection) you have to make things more complicate because you may want to protect against a managed debugger or a native debugger. Yes they're different.

Unless managed debugger is built on top of native one if you call native IsDebuggerPresent() you will always get FALSE for a managed debugger where Debugger.IsAttached would return true. Also opposite scenario is true: with a native debugger attached you will get TRUE from IsDebuggerPresent() but false from Debugger.IsAttached. In the big world you will meet all three types of debugger. For a better discussion about IsDebuggerPresent() you may read Anti-Debugging.

You may check both but you're still far from detecting debuggers because they only check for local managed/native debuggers but you don't have information about remote debuggers (unless you also call CheckRemoteDebuggerPresent()) or debuggers that don't live in user-mode (unless you also play with NtQuerySystemInformation). There are some slightly more robust techniques but you can't do it in managed world (see also Detecting System Debugger).

One possible solution is to debug your own process using DebugActiveProcess(), it you fail (and it's not a permission error) then another debugger is attached, moreover until you stay attached another debugger can't attach. Note that a process can not debug itself (AFAIK) then it must be done from a child process (which somehow communicate with main process you want to protect). It's nothing new, basically the same technique described in How to detect if the current process is being run by GDB? but Windows specific.

Read see also Debuggers aren't supposed to change behavior and Managed vs. Native debugging APIs for more information about this topic.