Search code examples
c#http-headersgooglebot

Format for X-Robots-Tag's unavailable_after

The info here - https://yoast.com/robots-meta-tags/ - states the following:

Note: Must be in RFC850 format (e.g., Monday, 15-Aug-05 15:52:01 UTC).

but the example from the grand daddy himself - https://developers.google.com/search/reference/robots_meta_tag - shows:

X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST

So, who is correct? Because I'm using C#, there is no easy way to create a DateTime if I have to cater to all the different time zones.

Solution

  • The specification for X-Robots-Tag is here. This is what it says about the date format for unavailable_after:

    The date/time must be specified in a widely adopted format including, but not limited to RFC 822, RFC 850, and ISO 8601.

    So, unfortunately, the format is not specified precisely, and you really will need to be able to parse multiple, potentially unknown, formats to write a crawler that works with all conformant tags.