Search code examples
amazon-s3azure-media-services

Azure Media Services Transfer Encoded Files to Amazon S3


Over at https://learn.microsoft.com/en-us/azure/media-services/media-services-deliver-asset-download it is stated "you can send media content to another application or to another content provider."

So if "another content provider" includes Amazon S3, this is exactly what I need. Basically, I want to upload a file to Azure Media Services, have the file encoded, and then have the resulting encoded files (plural) transferred (uploaded) to an S3 bucket.

The web page referenced earlier doesn't discuss the option further (it discusses downloading the files). Furthermore, the code sample seems to use the C# Azure SDK and I am able to use the REST API only.

Given the scenario, is transferring encoded output files from Azure to S3 possible, and if so, how can it be done?


Solution

  • It's not a first party feature in the AMS API at this time. The output of an Azure Media Services encoding job is a set of files in a container in Azure Storage. You have full access to the storage account, so any Storage API operations to move the file out of that storage account are supported.

    There are a couple of recommendations I would make for automating this movement of content out of Storage and over to S3.

    1) Azure Functions - you could write an Azure function that was triggered via WebHook from your Azure Media Services account. When the job completes, the Azure Function would need to get the Asset ID, get the storage account container URL and then do the HTTP copy to S3. Only problem with this route is that Functions do have a 5 minute operational limit, which could time out on you for very large file transfers. You can get around that by using the Function to put work onto an Azure Queue, and then you could have a pool (K8) of Docker Containers running in Azure Container waiting to do the long-running transfers based on the data in the queue. That would be more scalable and manageable long term.

    2) You could use one of the various open source projects on Github that are capable of copying from Azure Containers over to S3.

    3) (?) haven't tried this one myself - but I have seen references to using the Azure Storage Data movement library to do this, but I only see examples of coming in from S3, not going out. [EDIT - take this one out! I took a look and it does not support upload to S3]

    Finally, just so you know roadmap - we are working on adding support for ingest directly from S3 and upload to S3 in our Media Services API. It would be great to get your feedback on the requirements you have for upload to S3 (what authentication do you need, what restrictions, encryption support, etc.)