I am looking for guidance on dealing with an issue with websites I manage.
I have been having issues with sharing pages of my website learnagentformula.com and other websites that use this template.
Using the LinkedIn inspector had the following feedback:
Error We did not re-scrape https://learnagentformula.com/videos because the URL or one of its redirects is blocked by rules set in robots.txt file of URL domain or robots meta tags set in the http response..
My robots.txt is at https://learnagentformula.com/robots.txt
This started about 2 weeks ago for some of my sites and 1 week ago for others. I haven’t been able to find any info or changes on my end.
Tried posting different pages from my sites. It affects pages most of the time and once and awhile I works just fine.
I checked my robots.txt file and checked to see if others are having issues. LinkedIn directed me here.
You have the following line in your robots.txt file:
User-agent: *
Disallow: /*?
This targets all web crawlers, telling them not to scrape your site for any page with a query string; there are also more pages that you are disallowing below this line. Are those some of the pages you want LinkedIn to access? You can override this function specifically for LinkedIn by adding this code to your file:
User-agent: LinkedInBot
Disallow:
Which will allow the LinkedInBot to scrape all pages. You can individually specify any pages you want to disallow, and this will override your
User-agent: *
section.