Syncing is slow from persistent Disk to Google Bucket

Question

We have around 11TB of images in local storage and the same has been copied to Google Cloud Bucket. We have a requirement to sync all images incrementally i.e onlyn updated files. Currently we are syncing files using below gsutil command.

gsutil -m rsync -r -C /mnt/Test/ gs://test_images/test-H/

Issue which we are facing is it is taking around 6 days to copy and most of the time it is taking to scan the disk. Please let me know if any method to copy updated data at least for 24hours.

Answer 1

To increase the transfer speed, here some tips:

• Use regional storage, the closest to your VM
• Use a VM with at least 8vCPU to maximise the bandwith like described in quota

Depends on the machine type of the VM:

All shared-core machine types are limited to 1 Gbps.

2 Gbps per vCPU, up to 32 Gbps per VM for machine types that use the Skylake or later CPU platforms with 16 or more vCPUs. This egress rate is also available for ultramem machine types.

2 Gbps per vCPU, up to 16 Gbps per VM for all other machine types with eight or more vCPUs.