Search code examples
javaelasticsearchkairosdb

kairosdb and elasticsearch integration

I'm using Kairosdb as my primary db. Now I want to integrate the Elasticsearch functionalities to my data inside Kairosdb. As stated inside the docs I have to duplicate all entries of my primary db inside Elasticsearch database.

Update

What I mean is that, if I want to index something inside elasticsearch, I have to do, for example:

Retrieve data of Kairosdb, a example json {"name": "hi","value": "6","tags"}

and then put it inside Elasticsearch:

 curl -XPUT 'http://localhost:9200/firstIndex/test/1' -d '{"name": "hi","value": "6","tags"}'

If I want to search I have to do this:

curl 'http://localhost:9200/_search?q=name:hi&pretty=true'

I'm wondering if it is possible to not duplicate my data inside Elasticsearch, in a way which I can achieve this:

  • get data from KairosDB
  • index them using Elasticsearch without duplicate the data.

How can I go about that?

Solution

  • It sounds like you're hoping to use Elasticsearch as a secondary (and external) fulltext index for your primary datastore (KairosDB).

    Since KairosDB is remaining your primary datastore, each record you load into Elasticsearch needs two pieces of information (at minimum):

    1. The primary key field(s) for locating the corresponding KairosDB record(s). In the mapping, make sure to set "store": true, "index": "not_analyzed"
    2. Any fields which you wish to be searchable (in your example, only name is searched) "store": false, "index": "analyzed"

    If you want to reduce your index size further, consider disabling the _source field

    Then your search workflow becomes a two-step process:

    • Query Elasticsearch for name:hi and retrieve the KairosDB primary key field(s) for each of the matching record(s).
    • Query/return KairosDB time-series data using key fields returned from Elasticsearch.

    But to be clear. You don't need an exact duplicate of each KairosDB record loaded into Elasticsearch. Just the searchable fields, along with a means to locate the original record in KairosDB.