So I have a regular website and a blog in the same domain. In the future I plan on buying a domain exclusively for the blog but for now this is the way I'll do it.
The blog is in the directory /blog and there are no links from the main site to the blog. So I presume robots just can't get there.
Is there a way to send robots to a particular directory using robots.txt?
Is there any other solution than putting a link in the main site?
You probably want to use sitemaps to achieve that
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.