Search code examples
prometheusrancherthanos

Thanos Querier can only query data 2 hours ago

I followed this blog https://rancher.com/blog/2020/prometheus-metric-federation in setting up thanos in our kubernetes.

We have this set up where Cluster A contains only the - Thanos Receiver, Thanos Store and Thanos Query.

The data chunks are successfully loaded in the S3 every 2 hours. However, when we pointed the Grafana Dashboard to use the Thanos Query, no data results are shown if you query in range of 10 minutes (or less than 2 hours). But querying data with time range 2 hours ago and up, data are shown.

Is Thanos Querier can query real time data from the Thanos Receiver (from Prometheus)? This is my configuration for Thanos Querier deployment (below) Is there any configuration I have missed?

containers:
      - args:
        - query
        - --log.level=info
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.replica-label=prometheus_replica
        - --query.replica-label=rule_replica
        - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local
        image: quay.io/thanos/thanos:v0.15.0

enter image description here

Solution

  • Yes, Thanos Query is able to get real-time data from Thanos Receive. Thanos documentation for receive claims:

    It exposes the StoreAPI so that Thanos Queriers can query received metrics in real-time.

    Just add another --store option to your query invocation to point to thanos-receive gRPC port