Search code examples
monitoringgrafana-loki

Loki-distributed TSDB Indexes


I've got loki deployed in kubernetes in microservices mode and store chunks in GCS while using TSDB indexes - my questions are: do these indexes end up stored in GCS as well or do I need to setup persistent volumes for them? Do they get stored after a certain amount of time as they do with BoltDB (I read that they're stored after 15 min of flushing the chunks)? Is there a way to store them along with chunks in GCS?

I pushed logs to Loki, chunks got stored in GCS but after redeployment with some changes there were no logs in Grafana or labels in Loki anymore but the chunks were stored (this happened in a 10 minutes timeframe)

Thanks in advance and sorry for no config snippets, I'm writing this on the phone after work.


Solution

  • After more research I finally solved it.

    The indexes are uploaded in object store at 15 minute intervals while chunks are instantly uploaded after flushing.

    Also, the issue with not being able to query after 10 minutes was that after 10 minutes the ingesters flushed the chunks in storage and query_store_max_look_back_period should be 0 if not using filesystem as storage and mine was 730h.