There are usually some that I don't use in whatever project I'm working on (System.XML, System.XML.Linq for example).
Are there any drawbacks from leaving default assemblies that I won't be using in my project?
Unused referenced assemblies are removed by the compiler.
Create a new console app in VS and it will default reference inn several assemblies. Compile your empty program and open it in reflector, and you'll see that only mscorlib is referenced. The others are removed.
Same goes for unused using
statements. The compiler removes them.
You might want to remove unused references and using for the sake of keeping things clean and more readable.