Search code examples
database-designweb-crawlersearch-enginedatabase-schema

Database Design for Search Engine?

I have Created a app that crawl the website. here - http://crawler.javaignite.com/

I got all relative URL and having its meta keyword and description including title.

But now i want to store these data in some tables & i don't have idea about how search engine store data and search so help me create basic schema for this.

Solution

  • I have tried to create but need more improvement on this . basically i use only three tables

    table : website_mst

    website_id
website_url [root]
website_visitperiod  // no of days to revisit

    table: crawled_history

    contains multiple data as how many times it crawled

    crawled_id PK 
website_id FK [ref. website_mst]
crawled_url
crawled_datetime

    table: crawled_page
    contains only meta and description

    crawled_id  FK [ref. crawled_history]
meta_title
meta_keyword
meta_description
total_viewed
updated_on