Are there any approaches or architecture design patterns to implement secure, clean role/permission based access control and UI conveniences without coupling them together?
The long story.
I have seen ambiguous use of roles and permissions in many web applications and I have often experienced how these ambiguities have caused misunderstandings and implementation difficulties.
Here is a simplified example.
Business requirements say that permission set for some specific role should deny access to some part of the system that displays a full list of addresses. But at the same time, users of this role will need to read the addresses for an autocomplete list on some other web page.
I have seen how reckless developers create a permission entry to disable access to addresses, and later they discover that users actually need to read the addresses from other parts of the system. Then they invent another specific permission for special cases where addresses can be read.
But for me it seems ambiguous and potentially risky situation. If user has no access to some specific data, then he/she shouldn't be able to access it at all. Period. Adding a special permission just for dropdown lists seems like a deliberate security hole. If user loads the list through an async request and the server is using the same controller action to return the list (and it should - to avoid code duplication), then how the server will know when it should not return the addresses, if they are sometimes forbidden?
This situation raises the question: "why shouldn't users of some specific role see the full list of addresses in the first place, if they have access to the list through some other means?" And the answer I often get from business analysts is something like "Well, the address list is not forbidden for data security reasons, but just because users of this particular role are not expected to do anything with the address list and it would be redundant item in their workspace".
So, now the problem seems clear to me: some permissions exist just for controlling the UI and not strictly for controlling access to some data. Such (ab)use of permissions feels wrong to me. Therefore the question which was given at the very beginning.
Good writing! It pretty much feels that you already have your answer.
IMO user profiling and user access are not the same thing. Access rights should be handled as low level as possible (eg. if or not a user has read-access to a specific SQL table) and profiling in this case should only apply at the UI level ("what the user actually wants or needs to see").
When we talk about an application that has some kind of access right control, there's almost always some kind of an "engine" behind the UI that actually holds all the data. The WORST thing ever you can do is implementing the security anywhere else than the engine itself. The data must never be accessible in any other way than through that engine's own access control or otherwise it's not access control - it's UI restriction.
But that's the perfect world :/ In reality, like in all areas of work, software development also has been driven torwards being more and more cost-effective, agile and responsive to the client. Not surprisingly, this guides people to do fast and cheap decisions... like "hell, let's just make another SQL procedure that pulls the data out as an admin" instead of "we need to re-evaluate user access rights, and/or possible redesign our tables to keep consistency with the access privileges". It's always a short-term (bad) solution WHEN it's done, but some solutions are definitely more NO-NOs than others.
As a guideline I'd say that if you're not 110% sure what you're doing, it's a biggest NO-NO there is.
TL;DR: If some data should be accessible even in one place, it's not restricted by access control. If it's unneccessary to show accessible data somewhere, use user/application profiling for filtering it.