I have a server which can handle clients that are using my program. How can i know if a client (a smart client) had been edited my program and not uses the original program ? (for example he disassembled my program edited & compiled it for cheating purposes)
If by "using your program", you mean that the input your server gets comes from an un-modified version of something you distributed, all you can do is make it difficult for the somebody to cheat. You can never be 100% sure.
You could ask for a client program checksum to be sent with the request, but that can be dummied: it doesn't have to be the real checksum.
You could ask for a more complex check, up to a zero-knowledge proof, but you can't be sure that some client-side code has intervened to run the proof with your code, but then run modified code for other parts.
Worse case, you've armored everything, but it still has to run on a machine, so the attacher builds a custom virtual machine to mess with your head.
Instead, you might want to think about "What's good enough?". Once you know the key things you want to protect, figure out a way to test for those specifically. That would be a more specific question....