How do I debug Excel's loading of a COM server?

I have successfully written a 32-bit Excel RTD server. This is loaded in Excel fine, and typing =RTD(...) in a cell works. However, my 64-bit build of the same code doesn't work, and I'd like to know of any way to debug this. I can't find any relevant documentation.

I have a 64-bit RTD server DLL. Dependency walker doesn't show any problems with missing DLL dependencies, and in fact if I link a toy executable I can call into this DLL fine.

The DLL is successfully registered with regsvr32.

Excel even sees it but lists it as "inactive". In the past, while developing the 32-bit version this usually happened due to missing DLL dependencies (but no error message). As stated earlier, I can link to the DLL and dependency walker doesn't show any problems.

What else can I do to debug this problem? Were it open source I'd look at the Excel source code and try to do what it's doing but obviously that's not an option here.

The same code produces a 32-bit DLL that 32-bit Excel runs fine. But 64-bit Excel can't seem to use the 64-bit DLL.

Solution

The problem was the registry entries for the DLL. To answer my own question (and after learning more about COM than I ever wanted to), the best way is to write a COM client application and try instantiating the COM server in several ways. For RTD, below is a sample client app I used. If anybody has similar problems, I suggest first trying to use CoCreateInstance, then seeing if you can get the CLSID from the ProgId, then creating the instance using the ProgId, since that's what Excel does. Replace the UUIDs accordingly and "VCRTDServer.RTDFunctions" with whatever your ProgId is. The code:

/**
   Small test program to check that the COM DLL was properly
   registered
 */

#include    <objbase.h>

#ifndef RTD_ARCH
#    define RTD_ARCH 32
#endif

//
//Here we do a #import on the DLL ,you can also do a #import on the .TLB
//The #import directive generates two files in the output folders.
//
#import  "bin\\VCRTDServer.dll"

#include <iostream>
#include <stdexcept>
#include <string>
#include <tchar.h>
#include "IRTDServer_h.h"

using namespace std;


#define PROGID _T("VCRTDServer.RTDFunctions")
#define CLSID_STRING _T("{8D2EEA35-CBEB-49b1-8F3E-68C8F50F38D8}")

const CLSID CLSID_RTD = {0x8D2EEA35, 0xCBEB, 0x49B1,
                         {0x8F, 0x3E, 0x68, 0xC8, 0xF5, 0x0F, 0x38, 0xD8}};

const CLSID IID_RTD_UpdateEvent = {0xa43788c1, 0xd91b, 0x11d3,
                                   0x8f, 0x39, 0x00, 0xc0, 0x4f, 0x36, 0x51, 0xb8};

const CLSID IID_RTD = {0xec0e6191, 0xdb41, 0x11d3,
                       0x8f, 0xe3, 0x00, 0xc0, 0x4f, 0x36, 0x51, 0xb8};

static string GetLastErrorAsString();
static void run();

int main() {
    try {
        run();
        CoUninitialize();
        return 0;
    } catch(const exception& ex) {
        cerr << "Error: " << ex.what() << endl;
        CoUninitialize();
        return 1;
    }
}

void run() {
    cout << "CoInitializing" << endl;
    CoInitialize(nullptr);

    // if CoCreateInstance doesn't work, nothing else will
    // cout << "Checking CoCreateInstance" << endl;
    // IRtdServer *obj;
    // const auto hr = CoCreateInstance(CLSID_RTD,
    //                                  nullptr,
    //                                  CLSCTX_INPROC_SERVER,
    //                                  IID_RTD_UpdateEvent,
    //                                  (void**)&obj);
    // if(hr != S_OK)
    //     throw runtime_error("CoCreateInstance failed: " + GetLastErrorAsString());

    cout << "Converting prog id to clsid" << endl;
    CLSID clsid;
    const auto ptoc_res = CLSIDFromProgID(L"VCRTDServer.RTDFunctions", &clsid);
    if(ptoc_res != S_OK)
        throw runtime_error("CLSID error: " +  GetLastErrorAsString());
    cout << "Printing out class ID" << endl;
    cout << "Class ID: " << *((int*)&clsid) << endl;

    LPOLESTR progId;
    const auto progIdResult = ProgIDFromCLSID(
        CLSID_RTD, &progId);
    if(progIdResult != S_OK)
        throw runtime_error("Prog ID error: " +  GetLastErrorAsString());

     char buf[40];
     WideCharToMultiByte(CP_ACP, NULL, progId, -1, buf, 40, NULL, NULL);
     cout << "prog id is '" << buf << "'" << endl;

    cout << "Creating instance" << endl;
    RTDServerLib::IRtdServerPtr rtdServer;
    if(rtdServer.CreateInstance(CLSID_RTD) != S_OK)
        throw runtime_error("Could not create instance: " +
                            GetLastErrorAsString());

    cout << "Starting RTD server" << endl;
    const auto startResult = rtdServer->ServerStart(nullptr);
    cout << "Start result was: " << startResult << endl;
}



//Returns the last Win32 error, in string format. Returns an empty string if there is no error.
std::string GetLastErrorAsString() {
    //Get the error message, if any.
    const auto errorMessageID = ::GetLastError();
    if(errorMessageID == 0)
        return {}; //No error message has been recorded

    LPSTR messageBuffer = nullptr;
    size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
                                 NULL, errorMessageID, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, nullptr);

    std::string message(messageBuffer, size);

    //Free the buffer.
    LocalFree(messageBuffer);

    return message;
}