Compiler optimization on not used function parameters

Question

I had a weird problem for a couple days using Windows RPC. Now I found the solution but I want to know what is the right way to prevent this problem.

So basically I had a problem calling NdrClientCall2 function. As you can see in the docs, this function's third parameter is a pointer to the client-side calling stack. I changed the midl.exe generated C code to my own C++ version so my caller look like this:

void CppRpc::InternalCommand(int pSize, char buffer[1000000]) const
{
    RpcTryExcept
    {
        NdrClientCall2(PMIDL_STUB_DESC(&m_rpc_stup_description),
            PFORMAT_STRING(&MIDL_PROCFORMAT_STRING.Format[0]),
            reinterpret_cast<unsigned char *>(&pSize));
    }
    RpcExcept(1)
    {
        std::cerr << "Runtime reported exception " << RpcExceptionCode()
            << std::endl;
    }
    RpcEndExcept
}

In release mode using full optimization I got access violation for a day or two, then I understood that since parameter buffer is not used in this function compiler optimization will remove this parameter somehow.

However I need this parameter to be on stack so the NdrClientCall2 can stream it to the server. If I add any use for buffer for example char dummy = buffer[0]; to the function buddy then it will work fine.

So my question here is, how to prevent optimization on this case? I'm using visual studio 2017.

Answer 1

I'd not recommend doing this. Because since InternalCommand is your internal function, the compiler may change the layout of its parameters. Whereas (as I understand) you need specific layout where there's an int variable followed by the buffer reserved for the stack.

Not also that this layout may change if you change your function calling conventions (i.e. cdecl vs stdcall).

I would define a structure (POD) with a single int field, pass a pointer to it to your function, whereas the actual structure allocated by the caller has larger size.

Something like this:

struct StackFrame
{
    int pSize;
    char pStack[1000000]; // stack goes here
};

// ...

void CppRpc::InternalCommand(StackFrame* pStack) const
{
    RpcTryExcept
    {
        NdrClientCall2(PMIDL_STUB_DESC(&m_rpc_stup_description),
            PFORMAT_STRING(&MIDL_PROCFORMAT_STRING.Format[0]),
            reinterpret_cast<unsigned char *>(pStack));
    }
    RpcExcept(1)
    {
        std::cerr << "Runtime reported exception " << RpcExceptionCode()
            << std::endl;
    }
    RpcEndExcept
}