Search code examples
grafana-lokipromtail

Promtail: "error sending batch, will retry" status=500

when starting the promtail client, it gives an error: component=client host=loki:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6780207 vs. 4194304)"

promtail config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: 'http://loki:3100/loki/api/v1/push'


scrape_configs:
  - job_name: server-log
    pipeline_stages:
    static_configs:
      - targets:
          - localhost
        labels:
          job:  server-log
          __path__: /opt/log/*.log
          __path_exclude__: /opt/log/jck_*,*.log

I tried to start changing limits on the server, and run promtail with parameters:

/usr/local/bin/promtail-linux-amd64 -config.file=/etc/config-promtail.yml -server.grpc-max-recv-msg-size-bytes 16777216 -server.grpc-max-concurrent-streams 0 -server.grpc-max-send-msg-size-bytes 16777216 -limit.readline-rate-drop -client.batch-size-bytes 2048576

But judging by what I found, this is a grpc protocol error, or rather, in the size of the transmitted message, where the maximum is 4 mb

Solution

  • Changing the server parameters, not the client, solved the problem

    server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608

limits_config:
  ingestion_rate_mb: 15
  ingestion_burst_size_mb: 30
  per_stream_rate_limit: 10MB
  per_stream_rate_limit_burst: 20MB
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 744h
  max_query_length: 0h