Why System::Call is returning "0" when running 32-bit installer in 32-bit machine but when running it in 64-bit machine returning blank in NSIS

Question

I have created a 32 bit installer using NSIS and that should run in both 32-bit or 64-bit machines.

The below code is working fine in 32-bit machine where it is calling the DLL (Profile.dll) using System::Call and the corresponding function IsSplashScreenDisabled(). This function returns false and showing the Message Box. It is working as expected.

But the same installer when I run in 64-bit machine, System::Call is not returning "0" instead it is showing blank "". So I am not getting the message box.

And also If I change "$PROGRAMFILES32" to "$PROGRAMFILES64" then also it is showing blank "" and not showing the Message Box.

So Here I need your suggestions or thoughts for why the System::Call is working differently with 32-bit/64-bit installer and in 32-bit or 64-bit machines.

!include LogicLib.nsh

InstallDir $PROGRAMFILES32\MyAppTest

Page components
Page directory
Page instfiles

UninstPage uninstConfirm
UninstPage instfiles

Section
SetOutPath $INSTDIR
  File E:\TestNullSoft\Test.dll

System::Call 'KERNEL32::AddDllDirectory(w "$INSTDIR")'

 System::Call "$INSTDIR\Test.dll::IsSplashDisabled() i.r0 ?e" 
  Pop $1 
  ${If} $0 = 0
    
    MessageBox MB_OK|MB_ICONEXCLAMATION "Splash s Disabled.$\r$\nRolling back the installation..." IDOK

  ${EndIf}
SectionEnd


BOOL IsSplashDisabled()
{
    BOOL    bResult = FALSE;
    DWORD   dwSplashScreen(0);
    RegistryObj regObj(SETTINGS_REG_PATH);
    
    if (regObj.Get(SPLASH_SCREEN, testSplashScreen))
    {
        bResult = (BOOL) !testSplashScreen;
    }

    return  bResult;
}

Answer 1

The way System::Call parses the string has a flaw, if the string contains (..) it will parse that as the function parameters and it fails to load the .DLL and the call fails. On 64-bit Windows $ProgramFiles32 contains (x86) as part of the path and this triggers the flaw.

There are two ways to work around this:

A)

Use a combination of SetOutPath and AddDllDirectory to make sure the .DLL can be loaded from a relative path:

SetOutPath $InstDir
System::Call 'KERNEL32::AddDllDirectory(w "$INSTDIR")'
File MyFile.dll

System::Call 'MyFile::MyFunction()i.r0'

SetOutPath makes the path the current directory and AddDllDirectory adds the directory to the list of valid directories to load from.

B)

Manually load the .DLL and call the address directly:

SetOutPath $InstDir
System::Call 'KERNEL32::AddDllDirectory(w "$INSTDIR")'
File MyFile.dll

System::Call 'KERNEL32::LoadLibrary(t "$InstDir\MyFile.dll")p.r1'
${If} $1 P<> 0
    System::Call 'KERNEL32::GetProcAddress(pr1,m "MyFunction")p.r2'
    ${If} $2 P<> 0
        System::Call '::$2()i.r0'
    ${EndIf}
    System::Call 'KERNEL32::FreeLibrary(pr1)'
${EndIf}