I've a problem with high IO wait. I've tested some things with the following hardware and software
-physical machine (HP ProLiant), Debian 5, MySQL 5.1, hardware raid, ext3-fs
-virtual machine on VMware ESX 4 (HP ProLiant), Debian 5, MySQL 5.1, hardware raid, ext3-fs
-virtual machine on Xen 4, Debian 6, MySQL 5.5, software raid, ext3-fs
Now this happens: when copying some big files on disk, I can see a high wait rate in 'top' and other processes are going to be blocked. In example:
dd if=/dev/urandom of=./foo.txt count=1000 bs=1M (for creating a big file) cat foo.txt foo.txt foo.txt foo.txt foo.txt > foo.new (sending to /dev/null does only reading on disk)
while 'cat' is running I can't start another copy process. Also (and that's the biggest problem) INSERT statements running in mysql database are being blocked until they exit correctly. With 'innotop' I can see dozens of processes in mysql held on state 'freeing items' (in MySQL 5.5 this is state 'query end') until 'cat' ends. The blocking of INSERT statements also happens when doing a mysqldump. Should I do now no database backup anymore? All systems base on ext3 and scheduler is set to 'cfg'. I've tried it with 'deadline' and it seems to help a little bit, but not in all situtations. I still could not find out why!? And this problems seems to be so complex :-( It looks like massive writing causes this problem. Could it be a bad IO handling by scheduler?
I found a hint somewhere to change the linux io scheduler (cat /sys/block/[dev]/queue/scheduler). This helps a bit, but I believe, that there wouldn't be solution when some parallel tasks (or processes) have such a big different work load.