I am looking to run a pre-trained object detection model onto a folder of ~400k images which is about 1.5GB. When I've tried running locally, it was estimated to take ~8 days to complete (with keras yolov3). Thus, I am looking to use AWS SageMaker and S3.
When I have uploaded the zip folder of my images in the SageMaker jupyter notebook and tried to unzip by using bash command, an error pops ups saying that I have insufficient space. The volume assigned to my notebook is 5GB EBS, I do have other heavy datasets in my jupyter notebook space which could be causing this issue.
To tackle that, I am looking for a way where I can upload my data to S3 and run SageMaker to read the images hosted and run an object detection model over. However, it does not look like there's a method to unzip folders on S3 without using an additional service (read that AWS Lambda may help) as these services are paid by my school.
I could possibly re-run my code to extract my images from URL. In this case, how can I save these images to S3 directly in this case? Also, does anyone know if I am able to run yolov3 on SageMaker or if there is a better model I can look to use. Appreciate any advice that may help.
yes u are right u can upload thousands of images using aws cli using $aws s3 cp ; or $aws s3 sync