Using Charles Debugging with Chrome gives me 1-3% "Connection timed out"

Question

At my job, we use the debugging tool Charles Proxy a lot. So I always have it running in the background. Sometimes it's recording, other times it's not.
However, lately it has started acting up. For every web request I do, there is a 1-3% chance that it will fail to connect with the reason: "Connect: Connection timed out: connect"

Test case
I did a test with Charles: Use the 'repeat advanced' option on a file to fetch it 50 times:

As you can see, 2 of them failed. (And it took it 21 seconds to timeout). The 98 other times, it took ~4ms to complete the request.

I did the same test with Charles closed and Fiddler opened instead.
After 400 requests of the file, not a single one had failed.

Happens on normal usage
Here, it happened when I opened the company game site. A rather important request failed and thus, the site never fully loaded.

I have tried
I tried restarting my computer.
I tried updating to the newest version of Charles.
I tried uninstalling and installing Charles.
I tried disconnecting from the VPN we're on.

I tried closing down Charles and that seems to fix it. But I need to have Charles running and I don't want to open and close it all the time.
I tried using Firefox instead of Chrome and it seemed to happen very rarely or almost never. However Firefox is really bad to use when working with Flash.

Edit (Chrome net-internals)
I logged the output from chrome://net-internals/#events when trying to load a file that failed. Pastebin dumps: http://pastebin.com/dgMaLMKc then http://pastebin.com/dAGpJCJy

Answer 1

It would seem this was caused by the program Ad Muncher. After closing it down, the timed out connections has apparently stopped.