I'm trying to upload a large csv file of size 83.4 MB to a bucket in google cloud storage using the google-api-client gem in my rails4 app. My code to upload a file to a bucket in google cloud storage is as follows:
def upload_file_resumable(local_file_path,mime_type,relative_bucket_path)
media = Google::APIClient::UploadIO.new(local_file_path,mime_type)
bucket_name = APP_CONFIG[:google_storage_bucket]
file_name = relative_bucket_path.split("/").join("/") + "/" + File.basename(local_file_path)
resumable_result = client.execute(
api_method: api.objects.insert,
media: media,
parameters: {
uploadType: 'resumable',
bucket: bucket_name,
name: file_name,
},
body_object: {contentType: mime_type}
)
upload = resumable_result.resumable_upload
if upload.resumable?
client.execute(upload)
end
end
This works for files of small size but for a large file I get the following error trace:
2016-08-22T12:39:14.515Z 3710 TID-ouzo56edo WARN: execution expired
2016-08-22T12:39:14.515Z 3710 TID-ouzo56edo WARN: /Users/raza/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/openssl/buffering.rb:326:in `syswrite'
/Users/raza/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/openssl/buffering.rb:326:in `do_write'
/Users/raza/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/openssl/buffering.rb:344:in `write'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/session.rb:370:in `<<'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/http.rb:500:in `dump'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/http.rb:926:in `dump'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/session.rb:615:in `block in query'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/session.rb:613:in `query'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient/session.rb:164:in `query'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient.rb:1087:in `do_get_block'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient.rb:891:in `block in do_request'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient.rb:985:in `protect_keep_alive_disconnected'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient.rb:890:in `do_request'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/httpclient-2.4.0/lib/httpclient.rb:778:in `request'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/faraday-0.9.2/lib/faraday/adapter/httpclient.rb:36:in `call'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/faraday-0.9.2/lib/faraday/response.rb:8:in `call'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client/request.rb:163:in `send'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client/request.rb:174:in `send'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client.rb:648:in `block (2 levels) in execute!'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client.rb:645:in `block in execute!'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client.rb:636:in `execute!'
/Users/raza/.rvm/gems/ruby-2.2.2@views/gems/google-api-client-0.8.6/lib/google/api_client.rb:679:in `execute'
/Users/raza/workspace/views/app/services/google_cloud_storage_service.rb:15:in `upload_file_resumable'
The trace suggests that the error is occurring in this block of code
resumable_result = client.execute(
api_method: api.objects.insert,
media: media,
parameters: {
uploadType: 'resumable',
bucket: bucket_name,
name: file_name,
},
body_object: {contentType: mime_type}
)
How can I fix this ?
Thanks.
Maybe the upload is taking a long time, and your request handler times out. Try increasing the timeout or deadline setting(s).
To confirm that indeed a slow upload is the cause, run your function from the command-line with the large file, see it succeed, and measure how long it takes.