I've got several sub-websites that are basically html versions of powerpoint presentations. I need those files installed on every Azure web role of the parent application because the files MUST come from the same domain (there are dependent cross frame operations going on in a SSL environment.)
I would just add them to the project file and upload them, but that does not scale, these packages can be in excess of 5mb a piece, with a large number to exist eventually.
How could/should I think about pulling files from say, S3, or an Azure Blob, on startup of the role, and triggered manually by me when there is an update?
For the on-demand, the parent application is an c# MVC app, so I can use anything within that framework to trigger a job.
First: If you're running your site in Azure, you should consider Azure blobs vs. Amazon S3 storage, to avoid both egress costs (on AWS side) and latency.
Now: As far as grabbing content from storage, this is a fairly common pattern, since (as it looks like you've already seen) the deployment package grows pretty large. Plus, it's not as easy to update a single file, since you need to re-bundle / deploy the package.
In your web role, you have an OnStart() event which gets triggered prior to your instance getting added to the load balancer. This is probably the best place to place your content-download code (or in a startup script by triggering an init app or script to do the downloads). In C#, once you have a reference to a CloudBlockBlob, you can execute DownloadToFile() or DownloadToFileAsync() to grab content from your blob and store in the local web app's content directory. There are many other methods - just calling out the file-download ones as an example.
As far as auto-update: There's no concept of a blob-watcher; you'd need to implement something to recognize an update. Perhaps a Service Bus message on a Management topic, where each instance subscribes to this topic, listening for messages around updates? Then, every time you push new content, you can push a message to the topic identifying the content (maybe with the sub-website name). You can kick off a listener in each web role instance during the OnStart() as well.