Search code examples
amazon-web-servicesamazon-s3snowflake-cloud-data-platformapache-iceberg

S3 Intelligent - Tiering with snowflake managed iceberg tables


Trying to save on huge rarely used data storage costs, with snowflake managed iceberg tables. Would you apply the economical s3 intelligent-Tiering storage, to the iceberg’s external volume on s3, instead of the pricy s3 standard policy?

Obviously s3 would chill rarely used data files, which is fine as long as:

Cold storage data is still accessible normally with just sql whenever needed (no additional requests needed from the s3 service to heat up cold\frozen files first…). After cold data was heated-defrosted by someone, it remains so for a reasonable time (a few good days even if not addressed again), before going deep freeze again. Very much appreciate your thoughts on this matter! TIA Rea

Not tried anything yet just designing.


Solution

  • Looking at the S3 Intelligent Tiers against your requirements:

    • Frequent Access tier: Same as 'standard'
    • Infrequent Access tier: Cheaper if it is only accessed once per month
    • Archive Instant Access tier: Lowest tier you would want while remaining instantly accessible
    • Archive Access tier: You don't want to use this since it doesn't meet your need to be "accessible normally with just sql whenever needed"
    • Deep Archive Access tier: Even slower to get data, don't go there

    From How S3 Intelligent-Tiering works - Amazon Simple Storage Service:

    Activate the Archive Access and Deep Archive Access tiers only if your objects can be accessed asynchronously by your application. If the object that you are retrieving is stored in the Archive Access or Deep Archive Access tiers, you must first restore the object by using the RestoreObject operation.

    So, you should be fine as long as you don't use those last two levels.