Search code examples
testingiousabilityreliability

How you test your applications for reliability under badly behaving i/o

Almost every application out there performs i/o operations, either with disk or over network.

As my applications work fine under the development-time environment, I want to be sure they will still do when the Internet connection is slow or unstable, or when the user attempts to read data from badly-written CD.

What tools would you recommend to simulate:

  • slow i/o (opening files, closing files, reading and writing, enumeration of directory items)
  • occasional i/o errors
  • occasional 'access denied' responses
  • packet loss in tcp/ip
  • etc...

EDIT:

Windows:
The closest solution to do the job as described seems to be holodeck, commercial software (>$900).

Linux:
Open solution wasn't found by now, but the same effect can be achived as specified by smcameron and krosenvold.

Decorator pattern is a good idea. It would require to wrap my i/o classes, but resulting in a testing framework. The only remaining untested code would be in 3rd party libraries.

Yet I decided not to go this way, but leave my code as it is and simulate i/o errors from outside.

I now know that what I need is called 'fault injection'. I thought it was a common production-line part with plenty of solutions I just didn't know. (By the way, another similar good idea is 'fuzz testing', thanks to Lennart)

On my mind, the problem is still not worth $900. I'm going to implement my own open-source tool based on hooks (targeting win32). I'll update this post when I'm done with it. Come back in 3 or 4 weeks or so...

Solution

  • I didn't go writing my own file system filter, as I initially thought, because there's a simpler solution.

    1. Network i/o

    I've found at least 2 ways to simulate i/o errors here.

    a) Running a virtual machine (such as vmware) allows to configure bandwidth and packet loss rate. Vmware supports on-machine debugging.

    b) Running a proxy on the local machine and tunneling all the traffic through it. For the case of upd/tcp communications a proxifier (e.g. widecap) can be used.

    2. File i/o

    I've managed to deduce this scenario to the previous one by mapping a drive letter to a network share which resides inside the virtual machine. The file i/o will be slow.

    A cheaper alternative exists: to set up a local ftp server (e.g. FileZilla), configure speeds and use Novell's NetDrive to access it.