Search code examples
kubernetesgoogle-kubernetes-enginedocker-volumepersistent-volumesexpress-winston

Store Winston js log files on GKE

I'm using winston js for my node js app logging. I'm deploying my dockerized app on GKE and want to store my files outside the container. Where should I store those files and what path should be mounted.

I'm really new to the kubernetes volumes and can't find the right tutorial to follow.

Solution

  • There are multiple options to do or follow, now first thing why you want store logs in files ?

    You can just push the logs to the stack driver service of GCP and your logs will be stored over there, in that case, no need to use any volume or extra configuration required.

    Stack driver logging is managed service you can search the logs and push the logs from all the containers. In this way, your service will be stateless and you don't need to configure the volume for your deployment. Since container or POD is stateless running inside the cluster you can easily scale the application also.

    Still, if you are planning to use the volume there are plenty of option with below description:

    Use Node volume :

    In this container will create the logs files inside Node's volume on which it's running.

    Example : https://kubernetes.io/docs/concepts/storage/volumes/#hostpath

    Cons :

    • Logs will get removed as soon as Node is auto-scaled or removed from GKE cluster during maintenance or so.
    • Container can schedule to any node any time, or seq of logs might create the issue.

    Use PVC disk :

    • If you will use the Disk to store your logs for long persistent time like 30-40 days it will work.

    • You create volume which will be used by POD in K8s and POD can use this volume to create file and store and it won't get deleted from that disk unless you do it.

    Example : https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes

    Cons :

    • Only a single POD (Replica) can connect with this if you are using the access mode ReadWriteOnce, this can create issue if you are planning to do autoscaling or want to many replicas.

    NFS & EFS

    You have to use the ReadWriteMany access Mode if you are planning to store all replicas or PODs logs in K8s.

    For ReadWriteMany option you can use any NFS service or GCP EFS service.

    in this case all your PODs will write to the Single Volume NFS or EFS system and logs will be saved there, but the extra configuration is required.

    Example : https://medium.com/@Sushil_Kumar/readwritemany-persistent-volumes-in-google-kubernetes-engine-a0b93e203180

    Extra :

    The best option is to push logs to stack driver without during many configurations of logs and you can manage retention period over there. Just start pushing logs from the application container and you can scale replica seamlessly.