I have a website which I support as far as IE8, no further.
When I first launched the site, I decided to use CSS vendor prefixes for CSSs elements such as border-radius, box-shadow etc. I did this from a complete noob standpoint.
However, is a better approach not to use them and simply let browsers catch up rather than patch up for the sake of uniformity?
No, you shouldn't remove all of them, however you may as well remove the ones which are no longer required.
Can I use... is a great resource for checking browser support for various CSS, HTML and JavaScript features. If you perform a search for box-sizing, for instance, it will tell you that all modern browsers have partial support for this and that Firefox requires the -moz- prefix. You can also view all of the CSS support tables on just one page here.
There are a couple of online resources which display information about browser usage. An example of this is StatCounter. StatCounter offers browser version statistics which can be filtered on time. If we look at the last 3 months, we can guestimate that we should still aim to support Firefox 20+, Chrome 25+, IE 8+ and Safari 5.1+.