Error Loki distributed with Helm: no private IP address found, and explicit IP not provided

Question

I'm configuring loki-distributed on a kubernetes cluster via helm, but I'm getting the following error:

failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided. I found only one solution on the internet, which was on a forum about Grafana Tempo, in which I added the following snippet:

loki:
   extraMemberlistConfig:
     bind_addr:
       - ${MY_POD_IP}
backend:
   replicas: 2
   persistence:
     size: 1gi
     storageClass: nfs
   extraArgs:
     - -config.expand-env=true
   extraEnv:
     - name: MY_POD_IP
       valueFrom:
         fieldRef:
           fieldPath: status.podIP

However this configuration, some requests I get the following error: "too many unhealthy instances in the ring"

My current custom-values.yaml is:

loki:
  auth_enabled: false
  persistence:
    enabled: true
    storageClass: nfs
    size: 2Gi
  limits_config:
    max_global_streams_per_user: 10000
    max_query_series: 10000
    retention_period: 30d
  storage:
    bucketNames:
      chunks: loki
      ruler: test
      admin: test
    type: s3
    s3:
      accessKeyId: lokiaccess
      endpoint: minio.databases.svc.cluster.local
      insecure: true
      s3ForcePathStyle: true
      secretAccessKey: *****
  storage_config:
    boltdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/index_cache
      resync_interval: 5s
      shared_store: s3
    aws:
      s3: http://lokiaccess:******@minio.databases.svc.cluster.local/loki
      s3forcepathstyle: true
write:
  replicas: 2
  persistence:
    size: 1Gi
    storageClass: nfs
backend:
  replicas: 2
  persistence:
    size: 1Gi
    storageClass: nfs
read:
  replicas: 2
  persistence:
    size: 1Gi
    storageClass: nfs
table_manager:
  retention_deletes_enabled: true
  retention_period: 30d
monitoring:
  lokiCanary:
    enabled: false
  selfMonitoring:
    enabled: false
test:
  enabled: false

Has anyone experienced this and have a solution?

Answer 1

You can check the ring members via the API, for example if you do a port forward to loki-gateway, you can open these links on a web browser:

• Check the member list: http://localhost:8080/memberlist
• Check the datasource (what Grafana actually does for testing the DS): http://localhost:8080/loki/api/v1/labels

If your pods are not in the memberlist, make sure you add the extraArgs and extraEnv to each backend, write, read block.

Here's the catch: You need at least 2 write pods to not get the "too many unhealthy instances in the ring".