We are having issues with our SAN and we have been monitoring it's performance. One metric causing us much consternation is the Primary Write Cache Hit Percent. Specifically what ratio would be considered problematic?
We have hired two consultants and talked to the vendor and they are all giving us different numbers.
Your question is difficult to answer well, because you're lacking some detail.
However, I'd generally say - write cache hit rate should be 100%, or as near to it as possible. If you think about it - spinning disks are slow compared to processors and memory. Especially if you include RAID, where you'll need to calculate parity and write multiple times potentially.
We call that 'write penalty' - in order to write a RAID 5, you need to:
It therefore has a write penalty of 4 IOs needed per write. (RAID 6 has to do similar, but because there's dual parity has a write penalty of 6).
This turns your poor performance SATA drives into really dreadful performance. But there's a few tricks you can use. By coalescing writes, you can reduce the need to read back from the disk - if you can assemble a full stripe, you don't need to bother reading at all, because you have everything you need to calculate parity. So your write penalty drops significantly.
On a 4+1 RAID 5, you do 5 IOPs per 4 writes, giving a write penalty of 1.25 - which is actually better than RAID 1 would be, assuming you can 'full stripe write' every IO. This is particularly good for sustained sequential writes such as journals or transaction logs, but for 'general purpose' usage you won't always have that ideal scenario.
Likewise - write IOs can be deferred. Most IO patterns are 'bursty' - we want fast response NOW but our average is low. E.g. when you save a file - you want to write all of it to disk, and don't want to wait too long. But once you've finished saving, you don't need to do any more writing for a while.
So we use write cache - to even out the 'bursts' and give really fast initial response for the same throughput, and also to smooth out our RAID inefficiency. With that in mind - write cache hit rates should be high. 100% being the ideal target.
Anything less means that your incoming writes are too fast - on average - for your disks. Once write cache is full, incoming IOs have to be 'held' meaning really poor latency and system performance, because you lose your efficiency boost from full stripe writes, and you have to pay that write penalty immediately on your incoming write.
(There may be limited exceptions where some IOs bypass write cache, but this is generally the case).