Visual studio by default copies all dlls' to each project's bin folder. This also includes system dlls' (with the exception of mscorlib.dll and System.dll), such as System.Xml.Linq, System.ComponentModel.Composition (included since SL4) etc.
Since these files are included in each XAP, XAP sizes grow considerably. In my limited testing, setting "Copy local" to false doesn't seem to break anything. Why does Visual Studio add these files to the bin path? Since a user would already have Silverlight, can I assume that these assemblies are already installed in the GAC of each user or am I missing something?
The files you mention are not included in the runtime install of Silverlight. A Silverlight application that uses these libraries must supply them to the user as part of the download.
You can minimise some of the impact by setting the "Reduce XAP size by using library caching" option in the Silverlight tab of the project properties pages.
This option causes each of these files to be placed in separate Zip files that will be placed in the same folder as the XAP.
This helps in the scenarios where your site may have multiple Xaps for various applications or where you release new versions frequently. For a single app that changes rarely or is hit by many unique visitors its not so helpful.