Multiple processes writing corrupted files in Windows – writing 0’s into files

Question

We have a system composed of multiple processes running on two Windows 7 servers. These processes write to various files on multiple hard drives on the servers. Recently we had a very strange error. Many of the files we were writing started writing 0-valued bytes instead of good data. The sizes of the files look correct but the values are 0’s starting from a certain time.

This happened to a wide range of files and processes coming from very different code bases, some in C++ and some in C#. This rules out any simple bug in code which could cause this. This happened on different drives, which rules out that some drive hardware was at fault. We reran the system many times since and got this to reproduce only once. We can’t find a cause or pattern for this occurring.

We checked the Windows Events log but saw nothing suspicious there.

Given these facts, and through elimination, we think that one of the following is at fault:

1. OS software. It could be Windows or drivers or a Trojan virus, or some such software that has a system wide effect on other processes.
2. Hardware – but not disk drive hardware – but rather something common to all drives such as the motherboard.

Our questions:

1. Has anyone ever seen this phenomenon and knows what it might be?
2. Supposing we are able to reproduce this reliably, how can we go about studying these 0-writing processes?

Answer 1

It was silly of me to overlook this, but I have a certain answer for this now.

The reason the 0's got into these files was that we reset the servers with a hard reboot before properly closing open files or processes. Files that were open during this time got 0's inside them and those even "seeped" backwards into places in the file that were written much before the reboot, but, I suppose, not flushed to disk.