Search code examples
facebookseorobots.txt

Facebook and Crawl-delay in Robots.txt?


Does Facebook's webcrawling bots respect the Crawl-delay: directive in robots.txt files?


Solution

  • We don't have a crawler. We have a scraper that scrapes meta data on pages that have like buttons/are shared on FB.