How to Pass Negative Integer to COM/OLE Function?

Question

I am working on a Microsoft Word Add-In based on C++/ATL in Visual Studio 2010. I also am using the MFC-based COleDispatchDriver and supporting classes and used Visual Studio's ClassWizard to generate wrapper classes from the Microsoft Word type library. A sample wrapper for the Selection.Move function generated by the ClassWizard is below.

long Move(VARIANT * Unit, VARIANT * Count)
{
    long result;
    static BYTE parms[] = VTS_PVARIANT VTS_PVARIANT ;
    InvokeHelper(0x6d, DISPATCH_METHOD, VT_I4, (void*)&result, parms, Unit, Count);
    return result;
}

For functions like the one above, I have also written helper functions to take care of the VARIANT argument passing, as shown below.

long Move(int Unit, int Count)
{
      long result;
      static BYTE parms[] = VTS_PVARIANT VTS_PVARIANT ;

      VARIANT vaUnit;
      ::VariantInit(&vaUnit);
      vaUnit.vt = VT_I4;
      vaUnit.iVal = Unit;

      VARIANT vaCount;
      ::VariantInit(&vaCount);
      vaCount.vt = VT_INT;
      vaCount.iVal = Count;

      InvokeHelper(0x6d, DISPATCH_METHOD, VT_I4, (void*)&result, parms, &vaUnit, &vaCount);

      ::VariantClear(&vaUnit);
      ::VariantClear(&vaCount);

      return result;
}

When I call my function with a positive integer for the Count parameter, Word responds correctly, e.g. the following function call will move the selection "forward" (toward the end of the document) by one character.

m_oSelection.Move(1 /* wdCharacter */, 1);

However, if I try to move the selection one character "backward" (toward the beginning of the document) with the following function call, Word does not respond as expected.

m_oSelection.Move(1 /* wdCharacter */, -1);

It "seems" like the Word automation is treating the integer as an unsigned integer and my -1 value becomes 65535 causing the selection to jump forward. Inspecting the vaCount variant on the line with the InvokeHelper function call, the VS debugger shows the .iVal value as -1, but the "value" of the vaCount variant is being shown as 65535.

What am I missing to appropriately pass the negative integer as part of the COM function call?

Answer 1

The problem is that you are misusing VARIANT.

You are setting vaCount's vt field to VT_INT ¹ but are then assigning your int value to its .iVal field instead of its .intVal field. The .iVal field is a 16bit short that is used for VT_I2, whereas .intVal is a 32bit int used with VT_INT.

Likewise, you are setting vaUnit's vt to VT_I4 but then are assigning your int value to its .iVal field as well, instead of its .lVal field, which is a 32bit long.

^{1: why you are using VT_INT at all, instead of the more traditional VT_I4?}

Try this instead:

long Move(int Unit, int Count)
{
      long result;
      static BYTE parms[] = VTS_PVARIANT VTS_PVARIANT ;

      VARIANT vaUnit;
      ::VariantInit(&vaUnit);
      vaUnit.vt = VT_I4;
      vaUnit.lVal = Unit;

      VARIANT vaCount;
      ::VariantInit(&vaCount);
      vaCount.vt = VT_I4;
      vaCount.lVal = Count;

      InvokeHelper(0x6d, DISPATCH_METHOD, VT_I4, (void*)&result, parms, &vaUnit, &vaCount);

      ::VariantClear(&vaUnit);
      ::VariantClear(&vaCount);

      return result;
}

That being said, I suggest you use the CComVariant or _variant_t wrapper class instead of using VARIANT directly, and let it handle these kind of details for you. Also, because InvokeHelper() throws an exception on failure, so let the wrapper call VariantClear() for you when it goes out of scope:

long Move(int Unit, int Count)
{
      long result;
      static BYTE parms[] = VTS_PVARIANT VTS_PVARIANT ;

      CComVariant vaUnit(Unit);
      CComVariant vaCount(Count);

      InvokeHelper(0x6d, DISPATCH_METHOD, VT_I4, (void*)&result, parms, &vaUnit, &vaCount);

      return result;
}

long Move(int Unit, int Count)
{
      long result;
      static BYTE parms[] = VTS_PVARIANT VTS_PVARIANT ;

      _variant_t vaUnit(Unit);
      _variant_t vaCount(Count);

      InvokeHelper(0x6d, DISPATCH_METHOD, VT_I4, (void*)&result, parms, &vaUnit, &vaCount);

      return result;
}