F# interface with Tcl/Tk results in memory corruption

Question

I am writing an F# script that uses Tcl/Tk. Depending on how I call Tcl/Tk from F#, I am experiencing memory corruption. My best guess is that the memory corruption occurs because the F# garbage collector is moving the location of functions around when I allocate memory. Tcl/Tk obviously does not know about this movement and so bad things happen. I am trying to understand why one method produces memory corruption and ones does not.

The simplest code that shows the problem is below:

open System
open System.Runtime.InteropServices

// -----Tcl-----
// Change this literal to point to appropriate Tcl .dll
[<Literal>]
let TclDll = "c:/Tcl32/bin/tcl86.dll"

// Must have the MarshalAs attribute for objs:ObjPtr[] otherwise the array is assumed to be of size 1.
// The SizeParamIndex parameter specifies the int (the second argument) as being the length of the array.
[<UnmanagedFunctionPointer(CallingConvention.Cdecl)>]
type ObjCmdProc = delegate of nativeint * nativeint * int * [<MarshalAs(UnmanagedType.LPArray, SizeParamIndex=2s)>]objs:nativeint[] -> int

[<UnmanagedFunctionPointer(CallingConvention.Cdecl)>]
type CmdDeleteProc = delegate of nativeint -> unit

// 94 EXTERN Tcl_Interp * Tcl_CreateInterp(void);
[<DllImport(TclDll, EntryPoint="Tcl_CreateInterp", CallingConvention=CallingConvention.Cdecl)>]
extern nativeint tclCreateInterp()

// 180 EXTERN int Tcl_Init(Tcl_Interp *interp);
[<DllImport(TclDll, EntryPoint="Tcl_Init", CallingConvention=CallingConvention.Cdecl)>]
extern int tclInit(nativeint interp)

// 96 EXTERN Tcl_Command Tcl_CreateObjCommand(Tcl_Interp *interp, const char *cmdName, Tcl_ObjCmdProc *proc, ClientData clientData, Tcl_CmdDeleteProc *deleteProc);
[<DllImport(TclDll, EntryPoint="Tcl_CreateObjCommand", CallingConvention=CallingConvention.Cdecl)>]
extern nativeint tclCreateObjCommand(nativeint interp, string cmdName, ObjCmdProc proc, nativeint clientData, CmdDeleteProc deleteProc)

// 291 EXTERN int Tcl_EvalEx(Tcl_Interp *interp, const char *script, int numBytes, int flags);
[<DllImport(TclDll, EntryPoint="Tcl_EvalEx", CallingConvention=CallingConvention.Cdecl)>]
extern int tclEvalEx(nativeint interp, string script, int numBytes, int flags)

// -----Tk-----
// Change this literal to point to appropriate Tk .dll
[<Literal>]
let TkDll = "c:/Tcl32/bin/tk86.dll"

// 0 EXTERN void Tk_MainLoop(void);
[<DllImport(TkDll, EntryPoint="Tk_MainLoop", CallingConvention=CallingConvention.Cdecl)>]
extern void tkMainLoop()

// 118 EXTERN int Tk_Init(Tcl_Interp *interp);
[<DllImport(TkDll, EntryPoint="Tk_Init", CallingConvention=CallingConvention.Cdecl)>]
extern int tkInit(nativeint interp)

// -----Main program-----
// Initialize the Tcl/Tk interpreter
let interp = tclCreateInterp()
tclInit(interp) |> ignore
tkInit(interp) |> ignore

// Direct ObjCmdProc
let testMemDirect = ObjCmdProc (fun clientData interp objCount objs ->
    let mem = seq {for i in 0..1000000 -> i} |> Seq.toList
    printf "%A" mem
    0)

// Indirect ObjCmdProc (via createObjCmd)
let testMemIndirect (objs:nativeint []) (interp:nativeint) = 
    let mem = seq {for i in 0..1000000 -> i} |> Seq.toList
    printf "%A" mem
    0

// Creates an ObjCmdProc given a function
let createObjCmdProc fn = ObjCmdProc (fun clientData interp objCount objs ->
    fn objs interp)

// Ignore delete messages
let ignoreDelete = CmdDeleteProc ignore

// Create Tcl commands
tclCreateObjCommand(interp, "testMemDirect", testMemDirect, IntPtr.Zero, ignoreDelete) |> ignore
tclCreateObjCommand(interp, "testMemIndirect", (createObjCmdProc testMemIndirect), IntPtr.Zero, ignoreDelete) |> ignore

// Set up GUI
let evalStr = """
wm title . {}
ttk::frame .f
ttk::button .f.testMemDirect -text {Test Direct} -command {testMemDirect}
ttk::button .f.testMemIndirect -text {Test Indirect} -command {testMemIndirect}
ttk::button .f.exit -text {Exit} -command {exit}

grid .f
grid .f.testMemDirect
grid .f.testMemIndirect
grid .f.exit
"""
tclEvalEx(interp, evalStr, evalStr.Length, 0) |> ignore

// Start Tk
tkMainLoop()
0 // return an integer exit code

I can press the "Test Direct" button as many times as I want and there are no issues. I can press the "Test Indirect" button once without issue. But the second time I press it, I get:

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

The difference between the "Test Direct" button and the "Test Indirect" button is that "Test Direct" calls tclCreateObjCommand with an ObjCmdProc delegate while "Test Indirect" calls tclCreateObjCommand with a function that produces an ObjCmdProc delegate.

I am guessing that in the "Test Direct" case, F# is pinning the ObjCmdProc delegate and so there is no memory corruption. However, in the "Test Indirect" case, F# is moving the location of the delegate in response to garbage collection. (Note that this issue only shows itself when there is a large memory allocation - i.e., I am generating a large sequence and converting it into a list).

Assuming that my guess is correct (which it may not be), why is F# behaving like this? And more importantly, is there a way I can tell F# to not move the location of this function in memory? I have tried various things such as GC.KeepAlive and GCHandle.Alloc with no success.

---

Addendum:

To fix this problem, all I had to do is keep a reference to (createObjCmdProc testMemIndirect). So replace:

tclCreateObjCommand(interp, "testMemIndirect", (createObjCmdProc testMemIndirect), IntPtr.Zero, ignoreDelete) |> ignore

with:

let testMemIndirectRef = createObjCmdProc testMemIndirect
tclCreateObjCommand(interp, "testMemIndirect", testMemIndirectRef, IntPtr.Zero, ignoreDelete) |> ignore

Otherwise, F# thinks that function can be garbage collected. It is possible I might also have to change the ending code from:

// Start Tk
tkMainLoop()
0 // return an integer exit code

to:

// Start Tk
tkMainLoop()
GC.KeepAlive(testMemDirect)
GC.KeepAlive(testMemIndirectRef)
0 // return an integer exit code

to ensure the delegates are kept alive til the end of the program.

Note that at this point I am not concerned with the deletion of Tcl commands even though at some point I might be (as the answer to this question rightfully points out).

Answer 1

At the C level, the clientData parameter (which you've been setting to IntPtr.Zero) is how you'd pass around a pointer to some context that you need to correctly invoke a function that implements a command. Without it, you've just got a pure pointer to a function. The correct thing therefore is to find some way to pass the fn through that mechanism; you need to retain a reference to it until the registered cmdDeleteProc is called, and that callback's primary task is the dropping of that (owning!) reference (it'll be called whenever the command is deleted from the Tcl side of things). This is how Tcl's command implementation API is designed to work; you're strongly advised to work with it.

I think it is working in the case of the “direct” call because the reference from the variables in F# is keeping the function alive. You're fairly lucky it didn't blow up on you in that case as well, since it definitely still isn't safe.

---

I don't know how to pass a pointer to an F# function instance via a raw pointer. You might need to put it in a plain data structure of some kind and pass a pointer to that around. I also don't know how to tell the F# garbage collector that that data reference can't be collected until you say otherwise (i.e., until the Tcl deletion callback is done). But those are the things that you must do. You probably don't want to expose the details of this directly to most of your F# code; the connector library is something you're going to want to write just once and then have that deal with the details of integrating the .NET and Tcl memory management domains (which is basically classic C, except with a custom memory allocator for more speed).