How can a server, i.e. a remote host acting as a central service for multiple clients, detect malicious or invalid clients akin to Blizzard's Warden. In some way, these kinds of software ask a client for specific information every once in a while, which cannot be easily faked from a non-official client.
What I'm wondering is, how can such a mechanism be implemented so that it's hard or impossible to reverse engineer from the client side? Is there any such technique for open source client software (closed source server)?
Short answer: You can't. The client is fundamentally untrustable. Blizzard (and other purveyors of anti-cheat software) are engaged in a constant arms race with the cheaters. You can't just implement it once and be done with it; you have to constantly monitor your product (either heuristically or via player reports) for cheating, then figure out how to programmatically evaluate if someone is cheating.
The longer answer is that you keep your "secret sauce" detection off the client; the client instead just collects information, which it forwards to a trusted machine for analysis. This can make it harder for cheaters to avoid detection, since they only know what information is being collected, not what is being done with it. Eventually though, they'll figure out how to spoof that information, and your anti-cheat mechanism will need to then deal with that problem.
What you can do is implement heuristics in your server code to detect players who are sending inputs that should not otherwise be possible, and then flag those accounts for review or ban. This does nothing detect malicious software on a client, but it can detect the effects of that malicious software. So while you may not be able to pinpoint what is sending those invalid inputs, you can still act no the account.
More specifically to your question, though, it's impossible to give you examples, because you have to define what constitutes "cheating" in the context of your application, and then device methods for detecting it. This is a very domain-specific problem, and to make it more complex, you're unlikely to find open-source implementations of such systems, because they necessarily rely on obscurity to detect cheaters.