Search code examples
search-engineweb-crawler

What should be the initial list of urls for a crawler to start its work


I want a list of urls from where my crawler can start crawling efficiently so that it can cover a maximum part of web. Do you have any other idea to create initial index for different host. Thanks you


Solution

    • http://www.dmoz.org is a good seed.
    • As said before, to orient a crawl, querying a search engine gives good results.