Is it OK to do heavy processing in .NET's asynchronous callbacks, hogging them for multiple seconds before returning? Or am I depriving the OS / the runtime of important resources?
For example, consider TcpListener.BeginAcceptSocket. My callback starts off by invoking EndAcceptSocket, then spends a while receiving data, and only then closes the socket and returns. Is this how it was meant to be used, or am I expected to do the extra processing on my own thread?
Yes, this is how asynchronous Sockets (Tcp client, listeners, etc.) are designed for use. You should always ensure that you invoke the end aysnc method and then do whatever processing you desire. Not invoking the EndAccept(), EndSEnd(), EndReceive(), etc., etc. methods leaves you potentially open to a memory leak, so it's always a good practice to follow.
The threads that are used are no different than if you had manually spooled up a background thread yourself, and are infact desinged to be used for even "long term operations" of a couple of seconds. Trust me, you don't want anything that takes that long to be running on a dispatching or GUI thread.
I have over 90 mobile based systems which use asychronous Sockets for commucation to a Server and it does an excellent job: much faster than web services (remember all web protocols run on top of a Socket anyway), easy to detect errors, etc., etc.
I do the same on my Server code (mixed in with some other WCF for middleware and backend communication pieces) and it's the most scaleable comm that we've used. You'll have to do a Google on it but there's a guy who published his testing using this technology and he was able to support 1,000 concurrent communications to a server with just 11 threads. Not bad.
Server example from MS: http://msdn.microsoft.com/en-us/library/fx6588te.aspx
Client example from MS: http://msdn.microsoft.com/en-us/library/bew39x2a.aspx
It takes more than these to get it "perfected" but this is a good place to start.