After getting frustrated with Azure, I decided to try GCP. I wanted to try training a deep learning image classification model using GCP. To start off, I went to Cloud Storage through the Google Cloud Console UI and made a bucket to upload my images and... it freezes when I try to upload my image folder. Not a good first experience.
I've tried half a dozen times. I always get the little popup "upload started," but then it freezes. If I close it and reopen, the bucket will have made the folder and downloaded 3 or so images, but even if I let it run all night, it never makes progress beyond that. It just stops uploading and becomes nonresponsive. I have about 38,000 images, in total the folder size that I'm trying to upload is 2.5 GB.
There is no error message, and no sign that anything went wrong when I reload aside from the fact that only 3 or so images downloaded. Nothing else on my computer freezes. If I have other tabs open in the browser, they do not freeze. The tab with Google Console in it simply silently freezes. I tried Data Transfer to move the files to Google directly from Azure Blob Storage, but Azure wouldn't give Google permission to access the storage.
Can someone recommend a possible work-around to this? Should I be uploading my files some other way?
Whether or not a large number of files can be uploaded via the Console in-browser appears to be a bit sporadic. In my particular case, attempting to upload a large folder (not large files in a folder, a large NUMBER of files in a folder) at once results in the browser freezing due to memory problems in GCP's Cloud Storage AND in AWS's S3 when I try to upload it through their browser portals. I have tried this on Windows 10, on Linux Mint, and tried on both Google Chrome and Firefox to no avail.
Some GCP and AWS users report freezing or crashes when the number of files in the folder is large, but not necessarily when the size of the folder itself is large. This is my experience exactly--I can upload a large file or a folder with a few hundred files through Cloud Console, but not a folder with many thousands of files.
The only stable solution I've found is to surrender and abandon trying to upload a massive number of files in a folder via browser portals and instead upload via CLI. It isn't the solution I'd hoped for, but it is the only one I've found.
EDIT:
Since this correct (and only working) solution was downvoted for some reason,
See the comments to Kainax's answer to this question for multiple AWS users experiencing this problem.
See the question and answers to this question for the same issue I had in GCP.
Since this is an issue and not a feature, there is no official documentation on it.
Neither I nor anyone else as yet could solve this through the Cloud Console and this has been a recurring issue for years with some users. The only clear workaround is to avoid using the Console UI altogether when uploading a folder with a large number of files.