Validate out of process COM Server client is trusted

Question

I have an out of process COM Server that exposes a method I do not want called by code I do not trust.

Basically what I'm trying to do is prevent a clever customer calling these methods that I need access too. I respect that a different solution all together should be considered. But for reasons I rather not get into I don't really have that option. I can go with a less than robust solution, but would prefer to get as robust as possible.

I have the ability to change both the client and the server code. The server is written in C++ and the client is written in C#.

What's the best way to secure these calls?

I figured that such a problem would not be uncommon for interprocess communications or remote procedure calls. So I'm hoping there is an acceptable industry best practice, or maybe even a straight forward COM solution. In the absence of such, I'm leaning towards grabbing underlining data from the arguments and using pieces of that data as a string to encrypt using a key from another piece of the data and pass that to the server to be validated.

Answer 1

If there's an out-of-box COM solution, or best practice I didn't find any. One common technique I found was to use a hash key. The client generates a key and passes it to the server. The server generates the same key and if it doesn't match the client's key, the call is rejected.

I based my hash key on some secret that is extracted from the underlying data that the API's are acting upon. Since my data is different with each use the key differs every time. Both sides have access to the underlying data before the call so they can generate the same key. I choose to base64 encode mine, only because I preferred passing a NULL-terminating character array instead of a byte array to the server.

LPCSTR      lpszSomeSecret  = "Some Secret To Be Hashed" ;
std::string strEncodedHash  ; // the resulting hash. 

HCRYPTPROV  hProv           = NULL ;
HCRYPTHASH  hHash           = NULL ;
DWORD       cbSomeSecret    = (DWORD)::strlen(lpszSomeSecret) ;
BYTE*       pbHash          = NULL ;
DWORD       cbHash          = 0 ;
DWORD       cbHashSize      = (DWORD)sizeof(cbHash) ;
LPSTR       pchEncodedHash  = NULL ;
int         cbEncodedHash   = 0 ;

HRESULT hResult = S_OK ;

if (!::CryptAcquireContext(&hProv, NULL, NULL, PROV_RSA_FULL, 0)) {
    hResult = ::AtlHresultFromLastError() ;
} else if (!::CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
    hResult = ::AtlHresultFromLastError() ;
} else if (!::CryptHashData(hHash, (BYTE*)lpszSomeSecret, cbSomeSecret, 0)) {
    hResult = ::AtlHresultFromLastError() ;
} else if (!::CryptGetHashParam(hHash, HP_HASHSIZE, (BYTE*)&cbHash, &cbHashSize, 0)) {
    hResult = ::AtlHresultFromLastError() ;
} else if ((pbHash = (BYTE*)::malloc(cbHash)) == NULL) {
    hResult = E_OUTOFMEMORY ;
} else if (!::CryptGetHashParam(hHash, HP_HASHVAL, pbHash, &cbHash, 0)) {
    hResult = ::AtlHresultFromLastError() ;
} else if ((pchEncodedHash = (LPSTR)::malloc(
            (cbEncodedHash = ::Base64EncodeGetRequiredLength(cbHash, ATL_BASE64_FLAG_NOPAD | ATL_BASE64_FLAG_NOCRLF))
            )) == NULL) {
    hResult = E_OUTOFMEMORY ;
} else if (!::Base64Encode(pbHash, cbHash, pchEncodedHash, &cbEncodedHash, ATL_BASE64_FLAG_NOPAD | ATL_BASE64_FLAG_NOCRLF)) {
    hResult = E_FAIL ;
} else {
    strEncodedHash.assign(pchEncodedHash, cbEncodedHash) ;
}

if (pchEncodedHash) ::free(pchEncodedHash) ;
if (pbHash)         ::free(pbHash) ;
if (hProv)          ::CryptReleaseContext(hProv, 0) ;
if (hHash)          ::CryptDestroyHash(hHash) ;