Search code examples
webnetwork-programmingdnsweb-crawlertld

How do I obtain a copy of all dns records in existence (i.e. a list of all domain->ip pairs for every major TLD)?

I am trying to parse just the index page of every domain on the web. The only alternative that I can think of this that would produce somewhat acceptable results would be a general web crawl to identify every unique domain on the web.

Instead, I would like to be able to just iterate through a list of all domain names on the web. Does a public (or private but not incredibly expensive) database dump like this exist?

Related question that may answer this question: is it possible to host your own DNS server without querying any other dns servers?

I suppose that looping through every possible ipv4 combination is also an option but that would be less targeted than the set of all registered domains.

Solution

  • I think I may have found a solution: czds.icann.org/