Suppose I have an open source client and server software, both fully open source.
How can I make sure modified clients cannot connect to the un-modified server? Or detect modified clients? For example to prevent cheating in open-source massive multiplayer online games?
I cannot find anything on this subject and excepf for empty google searches I don't know where to start looking for information on this subject so I thought maybe someone here can answer this or even point me in the right direction to get the answer.
To do this perfectly would be either extremely difficult or impossible. You can add code to the client that will cause it to take a hash of itself and transmit that, then compare that hash server-side to known unmodified hashes, but a skilled programmer could easily fake a valid hash using such a method. You're better off making sure modifying a client doesn't grant an advantage when playing online.