Search code examples
apache.htaccesspdfsearch-engine

X-Robots-Tag not shown in HTTP response header

My sample PDF URL is: https://askanydifference.com/wp-content/uploads/2022/09/Difference-Between-Import-and-Export.pdf

I am trying to noindex all my PDF files in my wordpress website. While doing research, I learnt that they can only be marked noindex using .htaccess and not any other means as pdf files don't have any html code for meta tag.

So by following the solution given below, I added the X-Robots-Tag is my .htaccess file as:

https://webmasters.stackexchange.com/questions/14520/how-to-prevent-a-pdf-file-from-being-indexed-by-search-engines

<Files "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>


# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

<IfModule mod_headers.c>
Header set Content-Security-Policy "block-all-mixed-content"
</IfModule>

I have placed the .htaccess file in public_html folder.

But when I inspect the file in Firefox under Network tab, the X-Robots-Tag is not present.

All my caching is disabled

enter image description here

Any help on where I am going wrong

Solution 
  • <Files "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

    You have copied the linked solution incorrectly. To match a regex with the Files directive you need the additional ~ argument. ie. <Files ~ "\.pdf$">. (Although the FilesMatch directive is arguably preferable when using a regex.)

    However, you do not need a regex here. Just use the standard Files directive with a wildcard (not regex) pattern. For example:

    <Files "*.pdf">
:

    Reference: