copy multiple folders of similar names from google cloud to google colab

Question

I have a google cloud bucket which has 7 subfolders named subset0 to subset7. I want to copy all of them to google colab. Right now I am using code like

!gsutil -m cp -r gs://mybucket/datafolder/subset0 datafolder/

to copy each folder separately. I am not sure how I can write a for loop to copy all folders without repeating the same line 7 times. Thanks a lot!!

Answer 1

As @FerreginaPelona mentioned in the comments, you can use gsutil -m cp -r gs://mybucket/datafolder/subset* datafolder/ if your gs://mybucket/datafolder/ only contains subset0 to subset7 and no other subfolders.

However, if your source bucket path has other subfolders and you only want to specify your needed subfolders, you may put your subfolders in a list and use a for loop as shown below.

from google.colab import auth
auth.authenticate_user()

# Download the file from a given Google Cloud Storage bucket.
subfolder_list = ["subset0","subset1","subset2","subset3","subset4","subset5","subset6","subset7"]
for subfolder in subfolder_list:
  !gsutil -m cp -r gs://mybucket/datafolder/{subfolder} /datafolder