To explain the context, let us take example of a single threaded Chat server which makes use of asynchronous event notification (e.g. epoll etc) to handle IO and which makes extensive use of PKI and other Cryptographic tools during User Authentication and Registration processes which are handled and completed locally. While Message Indirection (from source-to-destination) is a gentle-on-CPU data-intensive process, few Cryptographic processes like signing of message using RSA key is computationally heavy and may become the slow path in the whole IO loop.
Can an attacker make use of this slow path to substantially degrade the performance of the server by making too many registration requests in a short duration? If yes then what are the methods to reduce the impact? If this is a real threat then how do large service providers manage it?
Let us expand the discussion to cover the Federated XMPP servers.
Really not an expert in this, but the following are common-sense reflexions might answer your questions.
This is why most registration forms use captchas.
Authentications aren't computationally heavy; they usually involve comparisons of (salted) hashes.
As @SergeBallesta pointed out, this is plain wrong; password hashing seems to be slow by design in order to prevent brute-force attacks. In fact, this post mentions the issue of vulnerability to DDoS, and suggests IP ban as a counter-measure (see paragraphs below and see also this thread for recommended number of rounds with BCrypt).
The number of authentication trials is usually limited using session parameters, and it doesn't seem unreasonable to deny authentication requests not associated with an existing session.
Massive authentication requests coming from the same IP in a short period of time can potentially be monitored and the IP banned temporarily. This would typically involve a monitor process independent of the application code itself.
I am not entirely sure here, I would probably test what I'm about to say if faced with this situation. I think that in the worse case, the overhead due to low-level encryption (eg SSL) is comparable to the application processing time, so I wouldn't worry about that.
Regarding message routing, you could potentially generate a new token every time an authenticated user sends N message(s), update the token validity at each submission and trigger an update when it expires. This might cause a slight overhead, but it allows to limit the rate of message submission per user, and therefore to control the overall server load due to message routing.
Hope this helps.