I've got a few clusters that have now been running for a month or so and am finding that temporary storage is being entirely gobbled up by Service Fabric log files. On a sweet fleet of F1 VMs where there is only 16GB of local storage I am just about out of space, a few of them are now down to 30MB, yes mega-bytes of storage (where less than 1GB is consumed by my application in all its versions).
In looking at the disk usage on the cluster VMs I can see clearly that the SvcFab\Log and SvcFab\ReplicatorLog folders are consuming over 90% of available space. Surely the SF can better handle this. Is there something I can toggle or configure to get it to flush some of it's data? Or better yet move it up to blob or table storage?
This must be an issue for others. What are others doing? And Service Fabric team, what is best practice for this?
So no useful help on this one. I resorting to tearing down that cluster and rebuilding it. Fortunately for me the cluster was one of a pair and I was able to simply redirect all traffic via TrafficMgr to the other cluster while I destroyed the faulty one and created a fresh one.
Pretty disconcerting to me. Had I not had this redundancy it would have been a rather huge problem. :-(