I am running 4 instances of MinIO in a k3s cluster, the servers operating MinIO have low processor usage and low ram usage they seem to be purring along like a fat kid eating cake.
All except one, this specific instance is struggling with put requests, time varies from 0.5 seconds to 17.5 seconds.
I have run several diagnostics including the deprecated
mc admin heal -r myminio
This returned some worrying stats, including 0.2% in red 0.1% in yellow All the other sites are 100% green.
So this points at disk being the issue but I dont believe this is conclusive. Since this should repair and certainly does not warrant 17 seconds put request time.
Finally the question:
What diagnostics can be run on these instances and is there some form of documentation around this. I have been all over the web, also tried posting in MinIO slack channel without luck.
Any assistance would be greatly appreciated.
Update:
persistent volumes are attached using nfs-subdir-external-provisioner
The underlying physical storage uses Synology Drive
After tracing, debugging and slamming my head on the desk for a while...
NFS share permissions were setup in such a way that asynchronous writes were disabled.
Nothing to do with MinIO nothing to do with Synology.
Incorrect setup.