My question is very straight forward: does any one experience increased build time when code contracts are enabled?
Yes, Code Contracts increases build times. There is work ongoing to improve it, but it will always take longer with Code Contracts than without.
The slowest part is the static checker, so you could enable that only for some builds or something... it depends on where the slowdown is causing you trouble.