i can't presist data in kubernetes volumes

Question

i have an application that record live traffic and replay them.

https://github.com/buger/goreplay

it is a simple app to use, but when i tried to use it with kubernetes i get a problem with persisting data in volumes.

i want to do this :

• in pod number one i use the goreplay container and other container that just have a simple python server... the job is the goreplay will listen to the requests coming from outside to the server and save them to a file , this is the deployment file :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: goreplay-deployment
  labels:
        app: goreplay-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: goreplay-app
  template:
    metadata:
      labels:
        app: goreplay-app
    spec:
      containers:
      - name: goreplay
        image: feiznouri/goreplay:2.0
        args:
          - "--input-raw"
          - ":3000"
          - "--output-file=requests_docker.gor"
        volumeMounts:
          - name: data
            mountPath: /var/lib/goreplay
      - name: myserver
        image: feiznouri/python-server:1.1
        args:
          - "3000"
        ports:
        - name: server-port
          containerPort: 3000
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: goreplay-claim

normally this will create the file.

the prblem is that when i delete the deployment, and create one that it's job is to read the file and forward the saving request to a server, it can't find the file , clearly i am using the volumes wrong , this is the second deployment that suppose to find and read the file :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: goreplay-deployment
  labels:
        app: goreplay-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: goreplay-app
  template:
    metadata:
      labels:
        app: goreplay-app
    spec:
      containers:
      - name: goreplay
        image: feiznouri/goreplay:2.0
        args:
          - "--input-file"
          - "requests_docker_0.gor"
          - "--output-http=http://localhost:3000"
        volumeMounts:
          - name: data
            mountPath: /var/lib/goreplay
      - name: myserver
        image: feiznouri/python-server:1.1
        args:
          - "3000"
        ports:
        - name: server-port
          containerPort: 3000
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: goreplay-claim

PS: this is the yaml file for the persistent volume :

apiVersion: v1
kind: PersistentVolume
metadata:
  name: goreplay-volume
  labels:
    type: local
spec:
  storageClassName: custum
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/data"

and this the file for the storage class :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: custom
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain
volumeBindingMode: Immediate

and this for the persistent volume claim :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: goreplay-claim
spec:
  storageClassName: custum
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi

how can i make this work and find and use the first file that i created in the second pod !

thank you in advance.

Answer 1

I replicated it and it looks like the volumes are fine.

What is not fine, is how you pass file paths to goreplay.

Here is what I did:

kubectl exec -it goreplay-deployment-899c49f95-7qdh4 -c goreplay sh
/home/goreplay # ps auxwf
PID   USER     TIME  COMMAND
    1 root      0:00 ./gor --input-raw :3000 --output-file=requests_docker.gor
   36 root      0:00 sh
   42 root      0:00 ps auxwf
/home/goreplay # ls /proc/1/cwd -l
lrwxrwxrwx    1 root     root             0 Feb 19 09:44 /proc/1/cwd -> /home/goreplay

Let me explain what you see here. I execed into goreplay container and checked the PID of goreplay process (PID=1). Next, I checked what is this process's current working directory by checking the /proc/1/cwd symlink. As you see it's symlinked to /home/goreplay.

What does it tell us?

It tells us that --output-file=requests_docker.gor is making goreplay to save the file in /home/goreplay/requests_docker.gor (since you are specifying path relative to process's current working dir instead of using absolute path pointing to volume). It should be set to:

--output-file=/var/lib/gorepath/requests_docker.gor

since it's the directory where the volume is mounted.

---

Same applies to the second deployment. You should specify:

--input-file=/var/lib/goreplay/requests_docker_0.gor`

so that it reads from the volume and not from the pod's home directory (/home/goreplay).

---

Change it and it should work.