I have an server engine, which generally works in the server (imaginatively). But occasionaly it will be executing on a client, or occasionally on another server, and will be wanting to use this server to do some of extra/additional processing.
The link between will be protected by ssl/tls and certificates. So the communication will be secure, but i'm not sure that calling engine is my code.
How would you go about athenticating that engine. What would Alice and Bob say on this subject?
You won't find a 100% way to do this.
Since:
Basically, since Karl can phone up Alice and pretend to be Bob, by having Bob's voice, and knowing everything Bob knows (the disassembled code), then there is no way Alice can verify that it really is Bob, or just a very good impostor.
If you design your software so that it can only run on specific types of hardware, with TPM or similar technology, then you might have a chance, but only through software, you cannot create a 100% solution.
Even with a TPM-enabled solution, you could still risk the impostor circumventing it by sitting inbetween.
It all depends on what kind of attacks you want to prevent.