This is just like this question only this is about the maven-assembly-plugin, not the javadoc plugin. I don't want to use a profile to do this because that would be more error prone than a command line option.
I figured this out on my own. You do it with -Dassembly.skipAssembly=true