I am trying to analyse time series in my DB using kibana and ES. First I indexed my data into ES using python API. Mapping which I used is:
data= {
"settings":{
"number_of_shards":1,
"number_of_replicas":1
},
"mappings":{
TYPE_NAME: {
"properties":{
"timestamp":{"type":"date", "format":"YYYY-MM-DD HH:mm:ss", "store":"true"},
"current":{"type":"float", "store":"true"},
"bid_qty":{"type":"float", "store":"true"},
"bid":{"type":"float", "store":"true"},
"offer":{"type":"float", "store":"true"},
"offer_qty":{"type":"float", "store":"true"},
"change":{"type":"float", "store":"true"},
"value":{"type":"string", "store":"true"}
}
}
}
}
Then I created an Index and dumped my data using the below code
es = Elasticsearch()
response = requests.put('http://127.0.0.1:9200/'+INDEX_NAME+'/', data=json.dumps(data))
row_data = ""
for row in rows:
row_data += '{"index":{"_id": "%s"}}\n' %row[0]
row_dict = {}
for i in range(1, len(row)):
row_dict[headers[i]] = str(row[i])
row_data += json.dumps(row_dict)
row_data += "\n"
response = requests.put('http://127.0.0.1:9200/'+INDEX_NAME+'/'+TYPE_NAME+'/_bulk', data=row_data)
after loading this data, when I try to add this index in kibana it asks that my timestamp field is of type data and then I click on create. But in the discover tab, I'm not able to find my data at all. Interestingly I make my timestamp field as a type string, and create a new index in ES, then all its content is getting loaded as non-time series data. But this is not of much use to me. Kindly suggest me if I'm doing anything wrong. THank you.
After keep on trying for a while, I got some part of it correct. So basically I was not giving the correct format. The "YYYY-MM-DD"
should be yyyy-MM-dd
. But then on visualizing data using kibana, in the time column, it showed comma separated two dates, one which was in my DB and the other one which was UTC. The major problem here was that all the graphs were using the UTC timings which I do not wanted. So after researching more, I used this format yyyy-MM-dd HH:mm:ss Z
and while inserting dates, I added +05:30 (IST) i.e. str(date) + " +05:30"
. After this, in kibana, in my time column, it showed two comma separated things, Invalid date, 2015-08-26 09:00:49
. However the graphs were made in my local time. I think I have found a temporary solution and if someone wants to improve this, is most welcome and put some more light in it. I think the correct solution should be time column in kibana should show only one date and that should be my local inserted date.