How do I return a unicode string / wstring / CStringW from a C++ dll to InstallScript?

Question

I am using InstallShield 2013 Premium. I created a C++ dll in Visual Studio 2010 to provide some functionality I could not achieve with InstallScript alone. My C++ function needs to return a small string (a username) to the InstallScript after doing considerable work to get this value.

Throughout the C++ am I using CStringW to represent my strings. Ideally, I would like to return it as Unicode, but I'm content with ANSI if that's my only option. I have tried numerous approaches with CStringW, std::wstring, std::string, LPCTSTR, LPSTR, char *... I tried direct returns, and attempts to return by reference. Nothing works!

Sometimes the dll function hangs, sometimes it throws an exception, at best it returns garbage values with non-printing characters. The official documentation on this does not seem accurate (it doesn't work for me!). Extensive Googling, and searching the Flexera boards produce "solutions" from others struggling with the same ridiculous problem, and yet non of those work for me either...

I didn't try this until the end, as I took for granted that you could pass strings between dlls and InstallScript easily enough. In retrospect, I should have started with the interface between the two and then developed the dll functionality after that.

Answer 1

Thanks for the help guys! I finally figured this out for myself though. There are multiple facets to the solution, however, which I have not found documented or suggested elsewhere.

Major points

1) return a WCHAR * from C++

2) use WSTRING as the corresponding return type in the InstallScript prototype

3) return it into a regular STRING variable in InstallScript, and treat it like any other

4) retain the value the WCHAR * points to in the C++ dll in a static variable, otherwise it apparently gets deleted and the pointer becomes invalid

If you've gotten far enough to find yourself in the same boat, I probably don't need to serve up every detail, but here's a chunk of example code to help you along:

---

Visual Studio Def File

LIBRARY MyIsDllHelper
EXPORTS   
   getSomeStringW   @1

---

C++ Header

#ifdef MYISDLLHELPER_EXPORTS
#define MYISDLLHELPER_API __declspec(dllexport) 
#else
#define MYISDLLHELPER_API __declspec(dllimport) 
#endif

#include <stdexcept>
#include <atlstr.h>

namespace MyIsDllHelper
{
    class MyIsDllHelper
    {
    public:         
    static MYISDLLHELPER_API WCHAR * getSomeStringW();
    };
}

---

C++ Source

#include "stdafx.h"
#include "MyIsDllHelper.h"

static CStringW someStringRetained;

CStringW getTheString()
{
    CStringW s;
    // do whatever...

    return s;
} 

WCHAR * MyIsDllHelper::MyIsDllHelper::getSomeStringW()
{   
    someStringRetained = getTheString();
    return someStringRetained.GetBuffer( someStringRetained.GetLength() ) + L'\0';
}

---

InstallScript

#define HELPER_DLL_FILE_NAME "MyIsDllHelper.dll"
prototype WSTRING MyIsDllHelper.getSomeStringW();

function DoSomething( hMSI )
    STRING svSomeString;    
    STRING svDllPath;   
begin

    // Find the .dll file path. (A custom function)
    GetSupportFilePath( HELPER_DLL_FILE_NAME, TRUE, svDllPath );    

    // Load the .dll file into memory.
    if( UseDLL( svDllPath ) != 0 ) then
        MessageBox ("Could not load dll: " + svDllPath, SEVERE );
        abort;
    endif;

    // Get the string from the dll 
    try         
        svSomeString = MyIsDllHelper.getSomeStringW();
    catch   
        MessageBox( "Could not execute dll function: MyIsDllHelper.getSomeStringW", SEVERE );
        abort;      
    endcatch;       

    // Remove the .dll file from memory.
    if( UnUseDLL( svDllPath ) < 0 ) then
        MessageBox ("Could not unload dll: " + svDllPath, SEVERE );
        abort;
    endif;

    // Use the string
    MessageBox( "svSomeString: [" + svSomeString + "]", INFORMATION );       

end;