Generally speaking, what are your recommendations on this? Currently takes close to 10 minutes for me to attach to a locally running IIS process hosting SharePoint 2007.
Make sure your symbol path includes a local cache directory so that it doesn't download symbols from Microsoft's public symbol server every time you attach.
Also, I haven't tried this with Visual Studio, but you may also be able to set up an exclusion list identifying modules for which you don't have symbols.