How can I upload images in bulk (about 100-200 images at a time) in Ruby on Rails?

Question

I want to upload images (around 200 kB each) in bulk. We have multiple options like as CarrierWave, Paperclip, and others. How can I perform these uploads in bulk?

Answer 1

TL;DR

Don't. Rails isn't optimized for bulk uploads, so do it out-of-band whenever you can.

Use FTP/SFTP

The best way to deal with large volumes of files is to use an entirely out-of-band process rather than tying up your Rails process. For example, use FTP, FTPS, SCP, or SFTP to upload your files in bulk. Once the files are on a server, post-process them with a cron job, or use inotify to kick off a rake task.

NB: Make sure you pay attention to file-locking when you use this technique.

Use a Queue

If you insist on doing it via Rails, don't upload hundreds of files. Instead, upload a single archive containing your files to be processed by a background job or queue. There are many alternatives here, including Sidekiq and RabbitMQ among others.

Once the archive is uploaded and the queued job submitted, your queue process can unpack the archive and do whatever else needs to be done. This type of solution scales very well.