Search code examples
c#amazon-web-servicesaws-sdkamazon-glacieraws-sdk-net

How to upload a file from my local machine to a vault of s3 glacier using c# in a console app?

did someone knows how to do that because i had investigate about, but i found only wrong/don't working answers I had try a lot of solutions but it seems to be wrong, like using the Chilkat directory , using ArchiveTransferManager ...

        Chilkat.Rest rest = new Chilkat.Rest();  
        bool bTls = true;
        int port = 443;
        bool bAutoReconnect = true;
        bool success = rest.Connect("glacier.eu-west-1.amazonaws.com", port, bTls, bAutoReconnect);
        Chilkat.AuthAws authAws = new Chilkat.AuthAws();
        authAws.AccessKey = ;
        authAws.SecretKey = ;
        authAws.ServiceName = "glacier";
        authAws.Region = "us-west-1";        
        success = rest.SetAuthAws(authAws);      
        rest.AddHeader("x-amz-glacier-version", "2012-06-01");            
        string filePath = "20190422.csv";
        Chilkat.Crypt2 crypt = new Chilkat.Crypt2();
        crypt.HashAlgorithm = "sha256-tree-hash";
        crypt.EncodingMode = "hexlower";
        string treeHashHex = crypt.HashFileENC(filePath);
        rest.AddHeader("x-amz-sha256-tree-hash", treeHashHex);
        crypt.HashAlgorithm = "sha256";
        string linearHashHex = crypt.HashFileENC(filePath);
        authAws.PrecomputedSha256 = linearHashHex;           
        rest.AddHeader("x-amz-archive-description", filePath);
        Chilkat.Stream fileStream = new Chilkat.Stream();
        fileStream.SourceFile = filePath;
        string responseStr = rest.FullRequestStream("POST", "/682988997959/vaults/streamqueuesvault", fileStream);
        if (rest.LastMethodSuccess != true)
        {
            Debug.WriteLine(rest.LastErrorText);
            return;
        }

        int respStatusCode = rest.ResponseStatusCode;
        if (respStatusCode >= 400)
        {
            Debug.WriteLine("Response Status Code = " + Convert.ToString(respStatusCode));
            Debug.WriteLine("Response Header:");
            Debug.WriteLine(rest.ResponseHeader);
            Debug.WriteLine("Response Body:");
            Debug.WriteLine(responseStr);
            return;
        }

        Debug.WriteLine("response status code = " + Convert.ToString(respStatusCode));


        string archiveId = rest.ResponseHdrByName("x-amz-archive-id");
        Debug.WriteLine("x-amz-archive-id = " + archiveId);

        string location = rest.ResponseHdrByName("Location");
        Debug.WriteLine("Location = " + location);
Solution

  • Here is a step by step guide on How to upload a file from my local machine to a vault of s3 glacier using c# in a console app?. First I would like to present some basic background information that will be used later in the solution. Feel free to skip ahead to the solution if you are smart on S3 Glacier.

    If you have AWS SDK for .NET and VS already installed, you can download the Repo from Github.

    Quick Intro to S3-Glacier

    Amazon S3 Glacier is Amazons low cost long term storage service.

    In Glacier terminology, an object is referred to as an Archive. Also the folders where you store archives are called Vaults. Its pretty simple - From the Glacier FAQ:

    Q: How is data within Amazon S3 Glacier organized? You store data in Amazon S3 Glacier as an archive. Each archive is assigned a unique archive ID that can later be used to retrieve the data. An archive can represent a single file or you may choose to combine several files to be uploaded as a single archive. You upload archives into vaults. Vaults are collections of archives that you use to organize your data.

    When you upload objects to S3 Glacier, the objects don't immediately appear in your Glacier console. Your Glacier console will refresh once a day.

    Amazon recommends you use the AWS SDK for .NET when developing C# applications that interface AWS services.

    Simple Solution

    Before you code, go into your AWS Console and create a S3 Glacier Vault name 'TestVault'.

    At the time of this solution (April 2019), I suggest you use Visual Studio 2019. These steps are similar for earlier versions of Visual Studio.

    The code I present was taken directly from the AWS SDK for .NET Documentation.

    Once your visual studio is ready, then follow these steps:

    1. Create a new project (use template -> Console App (.NET Framework) - not Console App (.NET Core) and name it ConsoleApp9

    2. Add the AWS SDK to your project via NuGet package manager command. Tools menu, select Nuget Package Manager, and click Package Manager Console. then type Install-Package AWSSDK.

      For a MAC use Project->Add Nuget Packages. Search for "AWSSDK.Glacier" and install it.

    3. Below is the working code. You need to copy most of this into your Program.cs and remove the default "Hello World" code. Your final Program.cs code should look like

      using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Amazon.Glacier;
using Amazon.Glacier.Transfer;
using Amazon.Runtime;

namespace ConsoleApp9
{
class Program
{

    static string vaultName = "TestVault";
    static string archiveToUpload = "C:\\Windows\\Temp\\TEST-ARCHIVE.txt";

    static void Main(string[] args)
    {

        try
        { 
            var manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USEast1);
            // Upload an archive.
            string archiveId = manager.Upload(vaultName, "upload archive test", archiveToUpload).ArchiveId;
            Console.WriteLine("Archive ID: (Copy and save this ID for use in other examples.) : {0}", archiveId);
            Console.WriteLine("To continue, press Enter");
            Console.ReadKey();
        }
        catch (AmazonGlacierException e) { Console.WriteLine(e.Message); }
        catch (AmazonServiceException e) { Console.WriteLine(e.Message); }
        catch (Exception e) { Console.WriteLine(e.Message); }
        Console.WriteLine("To continue, press Enter");
        Console.ReadKey();

    }
}
}

    4. Put the file that you want to be uploaded to Glacier as c:\Windows\Temp\Test-Archive.txt. You can put the file anywhere you want, just update the variable archiveToUpload in your code to reflect the location.

    5. If your region is not USEast1, Change the AWS Region on the line just after the try:

    var manager = new ArchiveTransferManager(Amazon.RegionEndpoint.YOUR-REGION);

    1. Run the program and it will upload the file. If you have installed the AWS SDK before this will likely work just fine and you will have a screen that shows your archive id.: enter image description here
    2. If you run into permissions or authorization errors - please follow these steps on setting up authorization for the AWS SDK. I recommend using a Credentials File (2nd option from top). Other problems could be wrong Vault Name or it cant find the file on your machine.
    3. When you go back to the Glacier console, you will not see any files uploaded. Glacier is low cost and slow moving compared to s3 and so your Vault contents are updated once a day.

    enter image description here

    As long as you get an ID in step 6, your file was successfully stored in Glacier.

    Hope this helps and you find success.