Let's say I want my clients to give the ability to create plug-ins for their application, but I don't want to make them hacks which poke with the memory of my program, is it possible to prevent this?
Or load the DLL in a kind of memory region where it won't have access to the main program memory?
You can let the plugins run in a separate process. Any information that is needed by the plugin is passed as a message to that process. Any result that is needed by the application is received as a message. You can have a separate process per plugin, or you can have all plugins run in the same process.
As an aside, most modern versions of a plugin feature use an embedded runtime environment, such as the JVM. Then, the plugin is running in the same process as the application, but within the confines of a virtual environment, which effectively limits the havoc the plugin can wreck upon your program. In this scenario, there is no DLL, but script code or byte code.