Regarding the spectre security issues and side-channel attacks.
In both x86 and ARM exists a method to disable caches/speculative access on specific memory pages. So any side-channel attack (spectre, meltdown) on these memory regions should be impossible. So why are we not using this to prevent side-channel attacks by storing all secure information (password, keys, etc.) into slow but secure (?) memory regions, while placing the unsecure data into the fast but unsecure normal memory? Accesstime on these pages will decrease by a huge factor (~100), but the kernel fixes are not cheap either. So maybe reducing the performance of only a few memory-pages is faster then a slightly overall decrease?
It would shift the responsibility of fixing the issues from the OS to the application-developer, which would be a huge change. But hoping, that the kernel will somehow fix all bugs seems not to a be good approach either.
So my questions are
Because our compilers / toolchains / OSes don't have support for using uncacheable memory for some variables, and for avoiding spilling copies of them to the stack. (Or temporaries calculated from them.)
Also AFAIK, you can't even allocate a page of UC memory in a user-space process on Linux even if you wanted to. That could of course be changed with a new flag for mmap and/or mprotect. Hopefully it could be designed so that running new binaries on an old system would get regular write-back memory (and thus still work, but without security advantages).
I don't think there are any denial-of-service implications to letting unprivileged user-space map WC or UC memory; you can already use NT stores and / or clflush to force memory access and compete for a larger share of system memory-controller time / resources.