Instead of doing the cleanup on the same thread (or launching a background thread and blocking till it completes) start the cleanup on a "background" (IsBackground = false, so it doesn't get terminated prematurely) thread and return immediately.
When is this a bad idea and how bad? Is this ever a good idea?
Replacing the Dispose() from the IDisposable with an asynchronous cleanup violates the Liskov Subsitution Principle as one would expect the resources to be available again immediately after the call.
I suppose this is some optimization needed because of frequent allocation/deallocation, which would mean that in the end you might just shift the problem to an increasing amount of objects pending to be disposed in the background thread. That will lead to memory shortage over a longer time and need some synchronization to make sure the amount of these objects doesn't grow to the sky.
Like Lazarus said, a more adequate solution could be to have a pool of reusable objects.