P/Invoke runtime STD error not redirected to file

Question

I am working with a C++ DLL that I access with P/Invokes from C# code. The problem is I cannot redirect the std error that happens in the unmanaged side to a file. This works well if the build is in DEBUG but when the build is in RELEASE the STD logfile-unmanaged does not contain the error but contains and does not close the application, it keeps running like there was no error:

Before Error
After Error

In C++ the STD error is redirected to a file like this:

extern "C" __declspec(dllexport) void RedirectStd()
{
    int fileNO = _fileno(stderr);
    _close(fileNO);
    int file = _open("logfile-unmanaged", O_CREAT | O_RDWR, 0644);
    _dup2(file, fileNO);
}

A runtime error in C++ is generated like this:

extern "C" __declspec(dllexport) void DoException()
{
    fprintf(stderr, "Before Error\n");
    int a = 0;
    int b = 10 / a;
    fprintf(stderr, "After Error\n");
}

In C# I'm calling both of those methods:

    [DllImport("TestError.dll", CallingConvention = CallingConvention.Cdecl, SetLastError = true)]
    internal static extern void RedirectStd();

    [DllImport("TestError.dll", CallingConvention = CallingConvention.Cdecl, SetLastError = true)]
    internal static extern void DoException();

My Main function:

    [HandleProcessCorruptedStateExceptions]
    [SecurityCritical]
    static void Main(string[] args) {

        Console.WriteLine("===== REDIRECT =====");
        Console.WriteLine("Native/Unmanaged exception redirected to logfile-unmanaged.");
        RedirectStd();
        Console.WriteLine("Native/Unmanaged std error redirected.");
        Console.WriteLine("");

        Console.WriteLine("===== EXCEPTION =====");

        DoException();

        Console.WriteLine("Waiting for program to crash");
        do {

        } while (true);
    }

EDIT:

C# console application: program.cs

using System;
using System.Runtime.ExceptionServices;
using System.Runtime.InteropServices;
using System.Security;

namespace ConsoleWithErrors {
    class Program {
        [DllImport("TestError.dll", CallingConvention = CallingConvention.Cdecl, SetLastError = true)]
        internal static extern void RedirectStd();

        [DllImport("TestError.dll", CallingConvention = CallingConvention.Cdecl, SetLastError = true)]
        internal static extern void DoException();

        [HandleProcessCorruptedStateExceptions]
        [SecurityCritical]
        static void Main(string[] args) {
            RedirectStd();
            DoException();

            Console.WriteLine("Waiting for program to crash");
            do {

            } while (true);
        }
    }
}

The console application stays opened like there was no error on the unmanaged side.

Answer 1

This works well if the build is in DEBUG but when the build is in RELEASE the STD logfile-unmanaged does not contain the error

It's probably optimized out, since you don't use the result of the division by zero. Try something like this instead:

extern "C" __declspec(dllexport) int DoException()
{
    fprintf(stderr, "Before Error\n");
    int a = 0;
    int b = 10 / a;
    fprintf(stderr, "After Error\n");

    return b;
}

This will break the result out of internal linkage and force the compiler to emit the code.

You seem to be new to C# P/Invoke however, so here's a few tips:

• the SetLastError attribute instructs the marshaller that the function will use the Win32 API SetLastError to set its error state, and your functions absolutely do not do so. You should remove it, because lying to the marshaller is never a recipe for success.
• the SecurityCritical attribute has to do with process elevation between trust levels. Again, your application has nothing to do with that and should be removed.
• the HandleProcessCorruptedStateExceptions attribute is deprecated since .Net Core (and including .Net 5). So if you're relying on it, you're nearing the end-of-life of your program, and it seems you barely even started writing it. The good news is that division by 0 isn't one of the exceptions that require this attribute to be processed, so again you should remove it!