Choosing the "right" web framework is quite challenging task, at least in Java we have a lot of them. But looking at JavaScript frameworks like AngularJS I doubt if we really need something heavy at server. Usually web framework is responsible for routing, templating, building pretty URLs and some other stuff. With AngularJS we can move all these responsibilities to client side. Then the backend becomes nothing more than REST listener and data validator. A thin layer between your application logic and client view. So why do we need web frameworks now if all we want is a REST listener?
At the moment I found two points which must be handled by server side: authentication/authorization and things requiring 'pushing' like Comet. Are these criterias enough to choose the "right" framework?
I'll give you one more capability that I've seen require back-end server support. It's those pages which generate a file. For example, they get a file from a third party and then hand it to the client as though they produced it directly, or they are generating a JPEG/PNG/GIF image on the fly, or perhaps a CSV/XLS dump of data. There may be ways to generate those on the fly from the front end and make them available for download, but sometimes the back-end is just easier for those.
Other than that, your assessment is 100% correct. You literally need less server for apps built with AngularJS than was needed with the previous request/response model of ASP/JSP/PHP/etc.
However, just because you need less doesn't mean you need nothing. Issues like data caching and how user sessions are handled can still come up even for smaller servers as you scale. But it has definitely opened things up for tech like Node.js to be considered that I would not have given much thought to a few years back.