How to get content of a kubernetes container log files in the Stackdriver in google cloud?

Question

I am new to kubernetes and google cloud and I need some help.

We have a pod with a single container runing in kubernetes in gke. There are logging messages that the container sends to its stdout and also some logging messages that it write into few log files in its storage.

The log messages sent to the container's stdout are picked by Stackdriver and we can see them there as expected. I want to get the messages written to the log files in stackdriver as well. My understanding from what I read here: (https://kubernetes.io/docs/concepts/cluster-administration/logging/#using-a-sidecar-container-with-the-logging-agent), is that a solution here is to add a sidecar container in the pod, and share a persistent volume between the two containers and somehow copy the log files in the shared volume and then make the sidecar container send the content of the shared log files to its own stdout (e.g. by sending a tail command to the sidecar container). Then those log messages will be picked by stackdriver, as they are in a container's stdout.

The problem is, how can I share the log files of my main container with the sidecar container. I tried to get the log files in the shared volume using a symbolic link (by adding a ln -s command to the first container), but then the sidecar container was not able to see the content of those files (although it was able to see the list of those files, I think because that would be only a shortcut to the storage of the main container, not a real copy of the files).

Another problem is, when I add a command to the main container (the ln -s command using command/args[]) in the template file where my pod is defined, then the default command of the container image will not be run! So I will not see the original logging messages of my main container in stackdriver anymore!

By the way, it seems even adding the sidecar container to the pod itself, disturbs the normal functionality of my main container. I assume this has to do with how I defined my sidecar container where I am probably missing something?

Thanks in advance for any advice!

Samanta

Answer 1

You don't need to create any symbolic links. It is enough to mount the volume in your main container so that it writes to mounted volume directly. The page you linked describes exactly that. Try using following snippet (source):

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-1
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-2
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}

Just use your own main container in place of the "count" container. Of course, if your application sends logs to a different directory than /var/log, you need to change the mountPath in the main container accordingly.