Search code examples
indexinghdfssearch-engine

Do search engines need to store the data?

Is it possible to create index with Solr or Elastic on static files stored in HDFS without duplicating the information?

Solution

  • With Solr you can use the RAMDirectoryFactory, which holds the full index in memory, without writing any files: https://cwiki.apache.org/confluence/display/solr/DataDir+and+DirectoryFactory+in+SolrConfig

    A quick search suggests that Elasticsearch has the equivalent with store type = memory, but I have no experience with that.