I'm running an AKS cluster on 3 nodes in different availability zones (for HA). There's an API running on it with pods on each cluster.
The plan is to add FileBeat as a DaemonSet (one pod on each node), and Logstash collecting the logs from each FileBeat instance. I chose DaemonSet over SidecarProxy pattern to consume less ressources on the node.
For FileBeat being able to read the logs from the API pods, I wanted to mount a volume on a managed azure disk on which the APIs can write their log files and from which FileBeat can read them.
The Azure Disk is of course only residing in one zone. So the problem is that the volume can't be attached if the node is not in the same AZ than the disk:
AttachVolume.Attach failed for volume "logging-persistent-volume" : Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: { "error": { "code": "BadRequest", "message": "Disk /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/<resource group>/providers/Microsoft.Compute/disks/<disk name> cannot be attached to the VM because it is not in zone '3'." } }
I'm quite new to Kubernetes and Azure. So, is there a way to have a shared volume for all API pods in this kind of environment?
Any help is appreciated! Thanks!
To answer your question:
You can add a storage solution in between which manages Azure Disks, and then create your volumes using that storage solution instead. An example would be Ceph, and you can use rook operator to set that up.
To solve your problem:
If you let your API log to stdout, kubernetes will write those log files in a disk at a specific location. Filebeat can then read from this location on each node and send your logs wherever you want them to be. This is the standard practice used for logging in the kubernetes environment, unless you have a specific need to write those logs to a volume, I wouldn't recommend that.
According to the filebeat docs:
You deploy Filebeat as a DaemonSet to ensure there’s a running instance on each node of the cluster.
The Docker logs host folder (/var/lib/docker/containers) is mounted on the Filebeat container. Filebeat starts an input for the files and begins harvesting them as soon as they appear in the folder.