ruby script to request videos and upload them to s3

Question

Problem: Transfer some videos from openstack(swift) to s3

Gems: fog, aws-sdk

I have an array of paths something like:

videos_paths = ["videos/attachments/5142/9f988f89ds9f8/lecture.mp4", "videos/attachments/3134/lecture2.mp4" ..... ]

I create urls for videos based on those paths.

My question is how can I "download" the video directly to S3 bucket and if there is any way to create a dir structure based on the video path.

E.g.

Video: https://myproject.com:443/v1/AUTH_a0fffc9ea361409795fb2e9736012940/production_videos/videos%2Fattachments%2F18116%2Fd6a5bd77a3b203cddsfb0c9d%2Foriginal%2Flecture.mp4?temp_url_sig=dce06f61775f24e88c80bed803b808668b073ed0&temp_url_expires=141243074

Workflow: Request video -> send it to S3 and store it in a similar dir structure

I accept any sugestion and ideas. If I can use other gem for this or if it can be done in another way.

Thanks,

I already checked:

1: Uploading Videos to S3 with Carrierwave and Fog

2: Upload videos to Amazon S3 using ruby with sinatra

Answer 1

Finally had time to finish this task before deadline :) If someone have a similar issue, I hope they can use something from this answer as inspiration.

#!/usr/bin/env ruby

require 'fog'
require 'aws-sdk'
require 'open-uri'

videos_paths = ["videos/attachments/5142/e01a339b41ce487643e85/original/lecture.mp4", "videos/attachments/5143/a4fa624f9324bd9988fcc/original/lecture-only.mp4", "videos/attachments/5144/95141978d5ecc14a1995fc/original/lecture.mp4", .... ] # 282 videos

fog_credentials = {
  "hp_access_key" => "",
  "hp_secret_key" => "",
  "hp_tenant_id" => "",
  "hp_auth_uri" => "",
  "hp_use_upass_auth_style" => true,
  "hp_avl_zone" => "",
  "os_account_meta_temp_url_key" => "",
  "persistent" => false
}

@storage = Fog::Storage::HP.new(fog_credentials) # Connect to fog storage
@my_time = 60 * 60 * 24 * 7 * 4 # 4 week links?

def make_temp_url(path, time = @my_time)
  @storage.generate_object_temp_url("videos", path, time, "GET")
end

def status(path, options = {})
  File.open('./stats.txt', 'a') { |file| file.puts "#{options[:msg]}: #{path}" }
end

s3 = AWS::S3.new(
  :access_key_id => '',
  :secret_access_key => ''
)

bucket = s3.buckets['']

videos_paths.each do |video_path|
  cur_url = make_temp_url(video_path)
  obj = bucket.objects[video_path]

  if obj.exists?
    status(video_path, msg: "Exists")
  else
    begin
      open(cur_url, 'rb') do |video|
        obj.write(video.read)
        status(video_path, msg: "Success")
      end
    rescue
      status(video_path, msg: "Error")
    end
  end
end