Since clients do not have access to the SQL CLR assemblies, it is only our own bugs that we have to fear. And with careful usage, "unsafe" can be quite safe. What do you think?
"Unsafe" isn't always a matter of who is executing the code, but what code is being executed and/or how it was coded. UNSAFE Assemblies allow for code that can, even in the ideal / proper usage, either destabilize SQL Server, or allow for security holes, or allow for odd / unexpected behavior. For example, using TimeZoneInfo to convert times between time zones requires unsafe even though it should be a somewhat simple calculation. The problem is that somewhere in that code base there is something that causes a memory leak. This has been experienced by folks who try to do bulk updates of date columns. UNSAFE is also used for code that could be SAFE but hasn't been verified by Microsoft, so it cannot be guaranteed. And in terms of using unsupported .NET Framework libraries, those are not only not guaranteed to work as expected (or not have memory leaks or be thread safe), but they aren't even guaranteed to work across any future .NET Framework update (case in point: ServiceModel became mixed-mode in .NET Framework v 4.0, so starting in SQL Server 2012, anyone who had imported ServiceModel into SQL Server 2005, 2008, or 2008 R2 could no longer import it and had to rewrite a bunch of code if they wanted to upgrade SQL Server).
But back to the question, how unsafe can it be when you control the code? You might not allow for any security holes, but you can definitely get yourself into situations of memory leaks and "odd" behavior due to shared memory (i.e. static variables) / synchronization issues that are really hard to reproduce and debug. For example, the following is a question on DBA.StackExchange about an error that started happening only when the system started calling the SQLCLR function(s) much more frequently. The problem was due to using a static variable to store stated, and then multiple sessions overwriting the values and getting unexpected values back when they went to read from that variable:
SQLCLR assembly throws error when multiple queries run simultaneously
Can UNSAFE Assemblies be used "safely"? Of course. If you are using a static variable for caching, and reload if it is ever null, that should be fine. Or, it is possible to load an unsupported .NET Framework library (which nearly always must be marked as UNSAFE) but only use "safe" methods within it (just because an Assembly has code that is "unsafe" doesn't mean that the "unsafe" code will ever be executed). The downsides there are: 1) you aren't aware of ALL of what those libraries are doing, even if you do check out some of it on reference.microsoft.com; and 2) you still face the possibility that at some point that library will be updated to be mixed-mode and then you have to rewrite everything.