How to call a x64 Assembly procedure in C#

Question

I am working on a project and currently have the following structure:

1. C# WPF project containing the User Interface as well as calls to external methods.
2. C++ DLL project containing an algorithm.
3. ASM DLL project containing an algorithm.

For simplicity, let's assume the algorithm simply takes no parameters and returns the sum of two, predefined numbers.

Here's the function signature and implementation in the C++ (second) project:

int Add(int x, int y)
{
    return x + y;
}

extern "C" __declspec(dllexport) int RunCpp()
{
    int x = 1, y = 2;

    int z = Add(x, y);

    return z;
}

And here's how I call the function in C#:

[DllImport("Algorithm.Cpp.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int RunCpp();

This works just fine - calling the function in C# returns the value 3, everything is working proplerly, no exceptions thrown.

However, I am now struggling to call the ASM procedure in C# code. I have seen (and tested myself to an extent) that it's impossible to call a MASM DLL directly in C# code. However, I've heard that it's possible to call ASM in C++ and call that function in C#.

1. My first question is - is calling ASM code actually possible directly in C#? When I try that, I get an exception that basically says the binary code is incompatible.
2. I have tried to use C++ to indirectly call the ASM DLL, and while I get no exception, the returned value is "random", as in, it feels like a remainder left in memory, for example: -7514271. Is this something I'm doing wrong, or is there another way to achieve this?

Here's the code for calling ASM in C++:

typedef int(__stdcall* f_MyProc1)(DWORD, DWORD);

extern "C" __declspec(dllexport) int RunAsm()
{
    HINSTANCE hGetProcIDDLL = LoadLibrary(L"Algorithm.Asm.dll");

    if (hGetProcIDDLL == NULL)
    {
        return 0;
    }

    f_MyProc1 MyProc1 = (f_MyProc1)GetProcAddress(hGetProcIDDLL, "MyProc1");

    if (!MyProc1)
    {
        return 0;
    }

    int x = 1, y = 2;

    int z = MyProc1(x, y);

    FreeLibrary(hGetProcIDDLL);

    return z;
}

Here, the code for calling C++ in C#:

[DllImport("Algorithm.Cpp.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int RunAsm();

And here's the ASM code of MyProc1, if needed:
Main.asm:

MyProc1 proc x: DWORD, y: DWORD

mov EAX, x
mov ECX, y
add EAX, ECX
ret

MyProc1 endp

Main.def:

LIBRARY Main
EXPORTS MyProc1

Answer 1

is calling ASM code actually possible directly in C#?

Example of this with two projects, C# and assembly based DLL. Looks like you already know how to get a C++ based DLL working. The project names are the same as the directory names, xcs for C# and xcadll for the dll. I started with empty directories and created empty projects, then moved source files into the directories and then added existing items to each project.

xcadll properties:

Configuration Type: Dynamic Library (.dll)
Linker | Input: xcadll.def

xcadll\xcadll.def:

LIBRARY xcadll
EXPORTS DllMain
EXPORTS Example

xcadll\xa.asm properties (for release build, /Zi is not needed):

General | Excluded From Build: No
General | Item Type: Custom Build Tool
Custom Build Tool | General | Command Line: ml64 /c /Zi /Fo$(OutDir)\xa.obj xa.asm
Custom Build Tool | General | Outputs: $(OutDir)\xa.obj

xcadll\xa.asm:

        includelib      msvcrtd
        includelib      oldnames        ;optional
        .data
        .data?
        .code
        public  DllMain
        public  Example

DllMain proc                            ;return true
        mov     rax, 1
        ret     0
DllMain endp

Example proc                            ;[rcx] = 0123456789abcdefh
        mov     rax, 0123456789abcdefh
        mov     [rcx],rax
        ret     0
Example endp
        end

xcs\Program.cs:

using System;
using System.Runtime.InteropServices;
namespace xcadll
{
    class Program
    {
    [DllImport("c:\\xcadll\\x64\\release\\xcadll.dll")] 
    static extern void Example(ulong[] data);

        static void Main(string[] args)
        {
            ulong[] data = new ulong[4] {0,0,0,0};
            Console.WriteLine("{0:X16}", data[0]);
            Example(data);
            Console.WriteLine("{0:X16}", data[0]);
            return;
        }
    }
}

For debug, use

    [DllImport("c:\\xcadll\\x64\\debug\\xcadll.dll")] 

xcs properties | debug | enable native mode debugging (check the box)