Search code examples
windowsvisual-studioassemblydllx86-64

Visual Studio debugger not working for a DLL build from assembly source files?


I'm currently using masm in a C++ wrapper to develop a .dll so it can be used by other applications, notably a c# one in Visual Studio 2022

However developing this way has some big drawbacks. Visual Studio has all sorts of little niggles about getting MASM working such as building properly if a file is 'included' in the asm source. Such files also cannot have break points. This means I have to resort to one large file, which is getting very unwieldy.

Its like Microsoft have given up supporting MASM, unless there's some way to build a DLL that allows debugging?

Or is there some better way to develop x64 .dlls on Windows, if there's no easy fix for the Visual Studio debugging problem?


Solution

  • Working example. I'm able to step through the c# code and into the assembly code. Directory for assembly based DLL is c:\xcadll.

    xa.asm:

            includelib      msvcrtd
            includelib      oldnames        ;optional
            .data
            .data?
            .code
            public  DllMain
    DllMain proc                            ;return true
            mov     rax, 1
            ret     0
    DllMain endp
            include xb.asm
            end
    

    xb.asm:

            public  Example
    Example proc                            ;[rcx] = 0123456789abcdefh
            mov     rax, 0123456789abcdefh
            mov     [rcx],rax
            ret     0
    Example endp
            end
    

    Program.cs - used to call the debug version of the dll:

    using System;
    using System.Runtime.InteropServices;
    namespace Program
    {
        class Program
        {
        [DllImport("c:\\xcadll\\x64\\debug\\xcadll.dll")]
        static extern void Example(ulong[] data);
    
            static void Main(string[] args)
            {
                ulong[] data = new ulong[4] {0,0,0,0};
                Console.WriteLine("{0:X16}", data[0]);
                Example(data);
                Console.WriteLine("{0:X16}", data[0]);
                return;
            }
        }
    }
    

    Project settings for DLL build:

    Create a .dll project for the assembly (or C or C++) code:

    project properties: Debug | x64
    

    properties for xa.asm:

    General / Excluded From Build: No
    Custom Build Tool / Command Line: ml64 /c /Zi /Fo$(OutDir)\xa.obj xa.asm
    Custom Build Tool / Outputs: $(OutDir)\xa.obj
    

    Project settings for C# build:

    Create a C# project

    project properties: Debug | x64
    

    As shown in the example code above, the C# program needs to import the debug version of the DLL.