NSIS System::Call - failing to invoke method

Question

I created my own dll that exposes one utility method, namely GenarateID. The header file looks something like this:

#ifndef GENERATE_ID_H
#define GENERATE_ID_H

#include <windows.h>
#include <string>
#include <fstream>
#include <stdio.h>
#include "openssl/sha.h"

int __declspec(dllexport) GenerateID(const char* fileName, char* retVal);

// some additional helper methods declared here, but omitted for brevity
// ...

#endif  // GENERATE_ID_H

What does this method do?
It reads the binary file passed as the first argument, gets some data, generates SHA256 hash of that data, and returns it through the second argument. On success, returned int value is 0, otherwise it's a negative value. One thing worth mentioning here is that I used OpesSSL library for this purposes.

Now, here is where the problem starts. As the title says, I'm unable to invoke my dll from NSIS using System::Call. (I created simple console test executable, that invokes the library with no problems.)

Here is what I tried, based on this short tutorial:

# copy the files
SetOutPath $INSTDIR
SetOverwrite on
File C:\Users\Strahinja\Desktop\CRX\testfile.zip  # copy the binary
File C:\OpenSSL-Win32\libeay32.dll                # copy OpenSSL library
File C:\Users\Strahinja\Desktop\CRX\IDGenerator\Release\IDGenerator.dll # copy my dll

# generate id
StrCpy $R0 "testfile.zip"
StrCpy $R1 ${NSIS_MAX_STRLEN}   ;assign memory to out parameter ?
StrCpy $R2 -1
#System::Call 'IDGenerator::GenerateID(t, t)i (r10, .r11).r12'
System::Call 'IDGenerator::GenerateID(t r10,t .r11)i.r12'
DetailPrint "File parameter: $R0"
DetailPrint "Generated ID: $R1"
DetailPrint "Return value: $R2"

NSIS says that returned value is error. Is there a way to check what's the error, and more importantly - am I using this right?

As a side note: I tried using an exe instead of a dll, copying it the same way into the same dir as dll, and calling it with:

nsExec::ExecToStack '"myidgen.exe"'
Pop $R2
Pop $R1

The executable here works fine, but I would still prefer the dll. I'm sensing that it might have something to do with the calling convention, or the dllexport (since it uses name mangling, AFAIK)
Any suggestions?

Answer 1

Error generally means syntax error but it could also mean that LoadLibrary or GetProcAddress failed. Use Dependency Walker to check if the exported function has a decorated name.

You might want to use a DEF file to export the function.

If you use EXTERN_C int __cdecl __declspec(dllexport) foo() it should also be exported without decoration but then you need to append ?c to the end of the system call...