That's been said that the BeginReceive and BeginSend callbacks are not executed on the .Net ThreadPool instead they are executed on the IOCP ThreadPool. For high performance servers it is critical for the IOCP threads to return to the pool as soon as possible which mean no heavy lifting within the callback of BeginReceive and BeginSend. What if I call an async method and await it within the callback.
Does it mean the IOCP thread will return to the threadpool and when the async operation will complete the callback method will continue on another available IOCP thread?
AFAIK, there's no guarantee that those callbacks will execute on IOCP threads.
If you make your callback asynchronous and use await, then the callback thread will be returned to the thread pool (whether it's an IOCP thread or a regular thread pool thread). Later, when the asynchronous method resumes, it will resume on a regular thread pool thread, not an IOCP thread (AFAIK this is undocumented but just makes sense).
However, I can't imagine a use case where you would actually want to do this. The code would be cleaner if you wrapped BeginReceive/EndReceive and BeginSend/EndSend within a task-based API and just used async/await throughout.