Protecting executable from reverse engineering?

Question

I've been contemplating how to protect my C/C++ code from disassembly and reverse engineering. Normally I would never condone this behavior myself in my code; however the current protocol I've been working on must not ever be inspected or understandable, for the security of various people.

Now this is a new subject to me, and the internet is not really resourceful for prevention against reverse engineering but rather depicts tons of information on how to reverse engineer

Some of the things I've thought of so far are:

• Code injection (calling dummy functions before and after actual function calls)
• Code obfustication (mangles the disassembly of the binary)

• Write my own startup routines (harder for debuggers to bind to)

  void startup();  
  int _start()   
  {  
      startup( );  
      exit   (0)   
  }  
  void startup()  
  {  
      /* code here */  
  }
  

• Runtime check for debuggers (and force exit if detected)

• Function trampolines

   void trampoline(void (*fnptr)(), bool ping = false)  
   {  
     if(ping)  
       fnptr();  
     else  
       trampoline(fnptr, true);  
   }
  

• Pointless allocations and deallocations (stack changes a lot)

• Pointless dummy calls and trampolines (tons of jumping in disassembly output)
• Tons of casting (for obfuscated disassembly)

I mean these are some of the things I've thought of but they can all be worked around and or figured out by code analysts given the right time frame. Is there anything else alternative I have?

Answer 1

What Amber said is exactly right. You can make reverse engineering harder, but you can never prevent it. You should never trust "security" that relies on the prevention of reverse engineering.

That said, the best anti-reverse-engineering techniques that I've seen focused not on obfuscating the code, but instead on breaking the tools that people usually use to understand how code works. Finding creative ways to break disassemblers, debuggers, etc is both likely to be more effective and also more intellectually satisfying than just generating reams of horrible spaghetti code. This does nothing to block a determined attacker, but it does increase the likelihood that J Random Cracker will wander off and work on something easier instead.