It causes my theme not to validate. And it's just plain wrong.
So why do they do it? And how can I fix it?
Concrete5 doesn't generally do anything to your markup, instead it's most likely due to the markup from a block you've added to the page (of course some of the blocks could be the default ones that came pre-installed with the system, which is somewhat of an arbitrary distinction I suppose).
As frz mentions, if you can say which block(s) the offending markup is coming from then specific advice can be given about how to address it (and you will definitely get more responses over at the concrete5 forums about this than here on SO).
All that being said, one of the great features of Concrete5 is the ability to customize the templates of each individual block, so it's very likely these can be addressed without having to hack the core system or anything drastic like that. Also I know that they're working on cleaning up the markup for the pre-installed blocks for the next version to be released in a month or two.
EDIT: For future viewers of this question, the problem turned out to be that the <?php Loader::element('header_required'); ?> line was placed AFTER the closing </head> tag in the OP's theme template (that line of code is responsible for outputting all system css and js files, so it needs to be placed inside your theme's <head> section).