How to upload large archives to Amazon Glacier using PHP and aws-sdk v3?

Question

This is my first time working with anything from Amazon. I am trying to upload multiple files into Amazon Glacier using the PHP SDK V3. The files will then need to be merged by Amazon into one.

The files are stored in the home directory of cPanel and will have to be uploaded via a cron job to Amazon Glacier.

I know I have to use the upload multi part method but I am not really sure which other functions it requires to make it work. I am also not sure if the way I calculated and passed the variables is correct.

This is the code I got so far:

<?php
require 'aws-autoloader.php';

use Aws\Glacier\GlacierClient;
use Aws\Glacier\TreeHash;

//############################################
//DEFAULT VARIABLES
//############################################
$key = 'XXXXXXXXXXXXXXXXXXXX';
$secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';   
$accountId = '123456789123';
$vaultName = 'VaultName';
$partSize = '4194304';
$fileLocation = 'path/to/files/';

//############################################
//DECLARE THE AMAZON CLIENT
//############################################
$client = new GlacierClient([
    'region' => 'us-west-2',
    'version' => '2012-06-01',
    'credentials' => array(
        'key'    => $key,
        'secret' => $secret,
  )
]);

//############################################
//GET THE UPLOAD ID
//############################################
$result = $client->initiateMultipartUpload([
    'partSize' => $partSize,
    'vaultName' => $vaultName
]);
$uploadId = $result['uploadId'];

//############################################
//GET ALL FILES INTO AN ARRAY
//############################################
$files = scandir($fileLocation);
unset($files[0]);
unset($files[1]);
sort($files);

//############################################
//GET SHA256 TREE HASH (CHECKSUM)
//############################################
$th = new TreeHash();
//GET TOTAL FILE SIZE
foreach($files as $part){
    $filesize = filesize($fileLocation.$part);
    $total = $filesize;
    $th = $th->update(file_get_contents($fileLocation.$part));
}
$totalchecksum = $th->complete();

//############################################
//UPLOAD FILES
//############################################
foreach ($files as $key => $part) {
    //HASH CONTENT
    $filesize = filesize($fileLocation.$part);
    $rangeSize = $filesize-1;
    $range = 'bytes 0-'.$rangeSize.'/*';
    $sourcefile = $fileLocation.$part;

    $result = $client->uploadMultipartPart([
        'accountId' => $accountId,
        'checksum' => '',
        'range' => $range,
        'sourceFile' => $sourcefile,
        'uploadId' => $uploadId,
        'vaultName' => $vaultName
    ]);
}

//############################################
//COMPLETE MULTIPART UPLOAD
//############################################
$result = $client->completeMultipartUpload([
    'accountId' => $accountId,
    'archiveSize' => $total,
    'checksum' => $totalchecksum,
    'uploadId' => $uploadId,
    'vaultName' => $vaultName,
]);
?>

It seems that the declaring of a new Glacier client is working and I do receive an UploadID, but with the rest I am not 100% if I am doing it right. The Amazon Glacier Vault where the files need to upload to and then get merged, remains empty and I am not sure if the files will only show ones the completeMultipartUpload has successfully been executed.

I also receive the following error when running the code:

Fatal error: Uncaught exception 'Aws\Glacier\Exception\GlacierException' with message 'Error executing "CompleteMultipartUpload" on "https://glacier.us-west-2.amazonaws.com/XXXXXXXXXXXX/vaults/XXXXXXXXXX/multipart-uploads/cTI0Yfk6xBYIQ0V-rhq6AcdHqd3iivRJfyYzK6-NV1yn9GQvJyYCoSrXrrrx4kfyGm6m9PUEAq4M0x6duXm5MD8abn-M"; AWS HTTP error: Client error: 403 InvalidSignatureException (client): The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'POST /XXXXXXXXXXX/vaults/XXXXXXXXX/multipart-uploads/cTI0Yfk6xBYIQ0V-rhq6AcdHqd3iivRJfyYzK6-NV1yn9GQvJyYCoSrXrrrx4kfyGm6m9PUEAq4M0x6duXm5MD8abn-M host:glacier.us-west-2.amazonaws.com x-amz-archive-size:1501297 x-amz-date:20151016T081455Z x-amz-glacier-version:2012-06-01 x-amz-sha256-tree-hash:?[ qiuã°²åÁ¹ý+¤Üª¤ [;K×T host;x-amz-archive-size;x-amz-date;x-amz-glacier-version;x-am in /home/XXXXXXXXXXXX/public_html/XXXXXXXXXXX/Aws/WrappedHttpHandler.php on line 152

Is there maybe a simpler way to do this? I do have full SSH access as well if that helps.

Answer 1

I think you misunderstood uploadMultipartPart. uploadMultipartPart means, you upload 1 big file, in multiple parts. and then do a completeMultipartUpload to mark that you have completed uploading one file.

from your code it looks like you are uploading multiple files.

it is possible that you do not actually need to use uploadMultipartPart

Maybe a you could use a regular "uploadArchive"?

ref:

https://blogs.aws.amazon.com/php/post/Tx7PFHT4OJRJ42/Uploading-Archives-to-Amazon-Glacier-from-PHP