I am maintaining a K8s cluster and recently kubelet evicted pods many times on different nodes because of disk pressure. After investigation, I found out that the problem is the container log files at /var/lib/docker/containers/.../*-json.log and these files can grow to hundreds of Gi and consume all of the disk.
I even face this when I was using a central logging stack consists of Kibana, Elasticsearch and Fluentbit. The fluentbit logs were around 500 Gi and after removing the central logging stack the disk pressure almost solved. But now I see it for some of my other components and its logs are consuming around 170 Gi.
What is some of the best practices and tools for managing log files in k8s?
Every Kubernetes installer should include Logrotate to handle this. https://kubernetes.io/docs/concepts/cluster-administration/logging/ has some basic info but it depends on your exact configuration.
EDIT: As I have now informed myself, Docker itself can also do log rotation directly so that's an option too.