Search code examples
textseocontent-typerobots.txt

robots.txt error : Content Type should be text/plain

I'm testing my site with the software called Search Engine Optimization (SEO) Toolkit 1.0 it displays this error :

The content type for the response from "htpp://mysite.com/robots.txt" is "text/html". The Web server should return "text/plain" as the content type for a Robots.txt file.

My robots.txt file is simply this : 

User-agent: * 
Allow: /

Saved with UTF-8 Without BOM Encoding.

Is this wrong?

What should be a default, harmless robots.txt file?

Thanks !

Solution

  • This is a MIME type issue and needs to be configured in your server.

    Here is a link: http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure

    For your specific hosting provider, they insert a small snippet of tracking javascript. Disable that feature following the customer service support in the comments and the mimetype should render.