Search code examples
google-colaboratory

Colab does not share storage on same account with different session

I created 2 Colab files on Google Drive. A few days before, they share mutual storage, when I create a folder on this file, I could see it on another file. But since yesterday, they did not like that anymore, they do not share storage.

I have tried to create 2 new files, or reset all the runtime, but the results still get the same.

Solution

  • Indeed, Colabs using the same Runtime Type used to run on the same VM, so it was possible to share files. However, this was a security breach. Indeed, consider the following scenario:

    • You run a private notebook on Colab and this writes secret data to disk.
    • Simultaneously, you run a public notebook written by a hacker, and it reads what's on the disk and sends it to the hacker.

    Moreover, sharing the same VM meant sharing the same GPU. Libraries such as TensorFlow grab all the GPU RAM when they start, so the first Colab would run fine while the second would have no GPU RAM left.

    Probably for these reasons, it seems that Google has decided to have a separate VM for each Colab, around August-September 2019 (but I haven't found an official announcement about this change). There seems to be a limit of 5 sessions running simultaneously for each Runtime Type, which means you can have up to 15 VMs running.

    Unfortunately, although this choice fixes the aforementioned problems, it does make it harder for Colab notebooks to communicate with each other. As @BobSmith suggested, one solution is to communicate through Google Drive. Here's how:

    First, run this at the beginning of each notebook (this will ask you to click on a link, authorize Colab to access your Google Drive, and give you an authorization code that you need to copy & paste back in Colab):

    from google.colab import drive
drive.mount('/content/drive')

    After a few seconds, you will get this message:

    Mounted at /content/drive

    Be sure to do this in both Colabs before you do anything else.

    Now look at the content of this folder:

    !ls /content/drive

    In my case, this outputs:

    'My Drive'

    Now whatever you write in the /content/drive/My Drive folder can be seen in both Colabs. For example, try running this on one Colab:

    !echo "Hello World!" > "/content/drive/My Drive/this_is_a_test"

    then run this on the other Colab:

    !cat "/content/drive/My Drive/this_is_a_test"

    It will output "Hello World!", showing that the Colabs can share data via Google Drive.