CreatePipe succeeds but does not generate a valid pair of handles

Question

I am writing some VBA code, and I want to be able to spawn a child process and communicate with it via stdin/stdout. I found a description of how to do this in C/C++ here:

Creating a Child Process with Redirected Input and Output

I am using Microsoft's VBA bindings for kernel functions as found here:

Office 2010 Help Files: Win32API_PtrSafe with 64-bit Support

Having started to port the C/C++ example to VBA, I have the code below, which only creates the pipe and nothing more. It checks that the operation succeeded and prints the handles that CreatePipe() outputs.

Sub test_pipe_creation()
    ' Security attribute structure
    Dim secAttrStdHand As SECURITY_ATTRIBUTES
    secAttrStdHand.nLength = Len(secAttrStdHand)
    secAttrStdHand.bInheritHandle = True
    secAttrStdHand.lpSecurityDescriptor = 0
    
    ' The handles
    Dim stdout_r As Long
    Dim stdout_w As Long
    stdout_r = &HDEADBEEF ' initialised to an obviously incorrect handle
    stdout_w = &HDEADBEEF

    Dim res As Long
    Dim e As Long
    
    Debug.Print "About to create pipe, handles currently have dummy values", Hex(stdout_r), Hex(stdout_w)
    
    ' Try to create the pipe
    res = CreatePipe(VarPtr(stdout_r), VarPtr(stdout_w), secAttrStdHand, 0)
    If res = 0 Then
        e = GetLastError
        Debug.Print "Failed to create pipe"
        Debug.Print "Error: ", e
    Else
        Debug.Print "Pipe created"
    End If
    
    Debug.Print "Pipe handle values are:", Hex(stdout_r), Hex(stdout_w)
    
End Sub

When I run this, I get the following output:

About to create pipe, handles currently have dummy values             DEADBEEF      DEADBEEF
Pipe created
Pipe handle values are:     DEADBEEF      DEADBEEF

So this means that CreatePipe() returned True indicating success, but it did not update the handle variables stdout_r and stdout_w.

To be sure that it wasn't just that the return code was wrong, I did also try checking the error code with GetLastError() regardless of the return code. The error value reported is 0 which indicates success, so Windows is definitely reporting that the operation succeeded.

What causes this behaviour? How can I create a pipe correctly from VBA code?

(Windows 11, Word, not sure which version of Word as it's not obvious how to check, but I just installed it recently along with the rest of Office 365)

Answer 1

By default, VBA passes parameters by reference, unless ByVal is specified (as you are doing in the nSize parameter).

VarPtr() returns a LongPtr, and you are declaring the phReadPipe and phWritePipe parameters of CreatePipe() as taking LongPtr by reference. So, you are actually passing CreatePipe() the addresses of the temporary LongPtrs that VarPtr() returns, not the addresses of your Long variables ¹ that VarPtr() is pointing at.

¹: which BTW, should be declared as LongPtr instead of Long, as Win32 HANDLEs are pointers, not integers.

That is why CreatePipe() is not modifying your variables. It is instead modifying the temporary LongPtrs from VarPtr(). You can easily verify that using explicit variables, eg:

Dim temp1 as LongPtr
Dim temp2 as LongPtr

...

temp1 = VarPtr(stdout_r)
temp2 = VarPtr(stdout_w)

Debug.Print "Pipe handle addresses are:", Hex(temp1), Hex(temp2)

res = CreatePipe(temp1, temp2, secAttrStdHand, 0)
' temp1 and temp2 get modified!

Debug.Print "Pipe handle addresses are:", Hex(temp1), Hex(temp2)

To solve that, try adding ByVal to the 1st two parameters, eg:

Declare PtrSafe Function CreatePipe Lib "kernel32" (ByVal phReadPipe As LongPtr, ByVal phWritePipe As LongPtr, lpPipeAttributes As SECURITY_ATTRIBUTES, ByVal nSize As Long) As Long

Alternatively, drop the VarPtr() altogether and just use ByRef instead:

Declare PtrSafe Function CreatePipe Lib "kernel32" (ByRef phReadPipe As Long, ByRef phWritePipe As Long, lpPipeAttributes As SECURITY_ATTRIBUTES, ByVal nSize As Long) As Long
...
res = CreatePipe(stdout_r, stdout_w, secAttrStdHand, 0)

Either way, CreatePipe() should now receive the addresses of your Long ¹ variables.