Search code examples
phpamazon-web-servicesamazon-s3aws-php-sdk

aws s3 php: fails to upload directory using UploadSyncBuilder

I am trying to upload a 14MB directory to s3 using aws php sdk. These are following scenarios and their results

  1. Upload from my local system(india) to s3(us-east-1): success
  2. Upload from ec2(ap-southeast-1) to s3(ap-southeast-1): success
  3. Upload from ec2(ap-southeast-1) to s3(us-east-1): FAIL

The following code I have used

require('application/classes/vendor/autoload.php');
use Aws\S3\S3Client;
use Aws\Common\Credentials\Credentials;
use Aws\S3\Sync\UploadSyncBuilder;
use Guzzle\Log\ClosureLogAdapter;
use Guzzle\Plugin\Log\LogPlugin;

$newCredentials = new Credentials("API_KEY", "SECRET_KEY");
$s3 = S3Client::factory();
//$logPlugin = Guzzle\Plugin\Log\LogPlugin::getDebugPlugin();
$stream = fopen('out.log', 'w');

$logPlugin = new LogPlugin(new ClosureLogAdapter(function ($m) use ($stream) {
    fwrite($stream, $m . PHP_EOL);
}), "# Request:\n{request}\n\n# Response:\n{response}\n\n# Errors: {curl_code} {curl_error}", true);

$s3->addSubscriber($logPlugin);

$s3->setCredentials($newCredentials);
$s3->setRegion("us-east-1");
UploadSyncBuilder::getInstance()
->setClient($s3)
->setBucket("BUCKET")
->setAcl("public-read")
->uploadFromDirectory("/mnt/1410503562Welcome to Leap!.zip_ext")
->setKeyPrefix("customs3/game10906/1410501964WelcometoLeap.zip_ext")
->setConcurrency(5)
->build()
->transfer();

Exception 

PHP Fatal error:  Uncaught exception 'Guzzle\Service\Exception\CommandTransferException' with message 'Errors during multi transfer (Guzzle\Http\Exception\RequestException) ./application/classes/vendor/guzzle/guzzle/src/Guzzle/Http/Message/Request.php line 569

Error completing request

#0 ./application/classes/vendor/guzzle/guzzle/src/Guzzle/Http/Message/Request.php(378): Guzzle\Http\Message\Request->processResponse(Array)
#1 ./application/classes/vendor/guzzle/guzzle/src/Guzzle/Http/Message/EntityEnclosingRequest.php(49): Guzzle\Http\Message\Request->setState('complete', Array)
#2 ./application/classes/vendor/guzzle/guzzle/src/Guzzle/Http/Curl/CurlMulti.php(286): Guzzle\Http\Message\EntityEnclosingRequest->setState('complete', Array)
#3 ./application/classes/vendor/guzzle/guzzle/src/Guzzle/Http/Curl/CurlMulti.php(244): Guzzle\Http\Curl\CurlMulti->processResponse(Object(Guzzle\Http\Message\EntityEnclosingRequest), Object(Guzzle\Http\Curl\CurlHandle), Array)
#4 ./application/classes/vendor/guzzle/guzzle in /vol/selfserve/application/classes/vendor/guzzle/guzzle/src/Guzzle/Service/Exception/CommandTransferException.php on line 25

I have setup log to debug and dig through Guzzle library as well. I found that when it fails, for a request no response comes. When I set concurrency to 1, it succeeds but takes much time. It fails even at concurrency of 2.

If any file fails, whole upload fails leaving it in partial uploaded state. I could not find any way to set options to retry if something fails.

Thanks for help!

Solution

  • After contacting with AWS support I was able to solve this with their help. Issue was in guzzle library. Following is the response from AWS support

    This error occurs when cURL fails to rewind a stream and does not associate an error with a curl handle. We've added checks to account for this situation and automatically retry. Please ensure that they are using the latest version of the SDK and version 3.9.2+ of Guzzle.