We would like to deploy a different robots.txt file to our staging and production boxes. We are using Microsoft Release Manager to automate the deployments and we use the .token capabilities to manage the web.config changes. This seems to break down in this case as we want to change the whole contents of the file and it does not allow us to put more than 1 line in a configuration variable. Example from staging where we don't want anything to be indexed:
User-agent: *
Disallow: /
Example from prod:
User-agent: *
Disallow: /assets/images/
Disallow: /3rdPartyFolder/
Sitemap: <url to site map>
Are there any capabilities for replacing multiple lines or entire files? Any other workarounds?
One option: Keep multiple versions of robots.txt in source control. When you deploy, rename the environment-appropriate version of the file to robots.txt.
Another option: Generate the robots.txt file via a PowerShell script that takes the environment as a parameter and adds appropriate entries to the file depending on the environment. This is probably overkill.