Search code examples
kubernetesgoogle-cloud-platformgoogle-compute-enginegoogle-kubernetes-enginejupyterhub

How to migrate data from Google Cloud VM to Google Kubernetes Engine?

I am running JupyterHub on Google Cloud VM but due to some reasons I am not able to access JupyterHub running on VM now. Rather than resolving the issue with current JupyterHub I wanted to migrate JupyterHub on our Google Kubernetes Engine, so I installed another JupyterHub on Google Kubernetes Engine using zero-to-jupyterhub-k8s.

Now everything is running fine but I want to migrate the data saved on the old JupyterHub VM to my new JupyterHub. The new JupyterHub using Persistent Volume claims as storage for each of the pods of users. Could someone please let me know how can I do it?

Solution

  • Posting this answer as a community wiki for a better visibility as well to add some additional resources that could help when encountered with similar scenario.

    The issue portrayed in the question was resolved by copying user data to GCS bucket and then mounting the data to the user pods as posted in the comment:

    I solved this issue by copying the data from the VM to Google Cloud Storage and then mounted the GCS Bucket on the user pods in JupyterHub on Google Kubernetes Engine.

    The guide for installing zero-to-jupyterhub-k8s:

    Resources on mounting GCS bucket to the Kubernetes pod:

    Citing the Github page:

    Disclaimer!

    The big catch is that for this to work, the container has to be built with gcsfuse. The Dockerfile includes a base build for debian jessie.

    The most note worthy parts of the configuration are the following:

    securityContext:
 privileged: true
 capabilities:
   add:
     - SYS_ADMIN

    For the container to have access to /dev/fuse it has to run with SYS_ADMIN capabilities.

    lifecycle:
 postStart:
   exec:
     command: ["gcsfuse", "-o", "nonempty", "test-bucket", "/mnt/test-bucket"]
 preStop:
   exec:
     command: ["fusermount", "-u", "/mnt/test-bucket"]