OpenTelemetry Collector Version: 0.82.0
Elastic APM Server Version: 7.12.1
HAProxy Version: 2.0.31-0ubuntu0.2
APM Server's service port is 8080 and it's in insecure mode. I have a HAProxy LB configured in front of more than one APM Server.
Strangely, when I set exporters.otlp.endpoint to the APM Server instance IP, traces are collected normally, but when I set it to the HAProxy's IP, I get an *error.
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: 2023-09-05T14:17:00.601+0900 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to http://haproxy-apm-lb:8080/v1/traces responded with HTTP Status Code 404", "dropped_items": 8}
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:391
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:126
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:195
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:47
[OTELCOL-SERVER-001]
# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
otlphttp:
endpoint: "http://haproxy-apm-lb:8080"
tls:
insecure: true
headers:
Content-Type: "application/json"
logging:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging,otlphttp]
...
# /etc/otelcol/config.yaml - working properly
...
exporters:
otlp:
endpoint: "http://elastic-apm-server-001:8080"
tls:
insecure: true
headers:
Content-Type: "application/json"
logging:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging,otlp]
...
# /etc/otelcol/otelcol.conf
OTELCOL_OPTIONS="--config=/etc/otelcol/config.yaml"
OTEL_EXPORTER_OTLP_HEADERS="Content-Type=application/json"
OTEL_EXPORTER_OTLP_PROTOCOL="http/json"
[HAPROXY-APM-LB]
# /etc/haproxy/haproxy.cfg
...
frontend http-in
bind *:8080
default_backend http_main
acl acl_main path_beg /v1
http-response set-header Content-Type application/json if acl_main
use_backend http_main if acl_main
backend http_main
mode http
option httpchk GET /
http-request add-header Content-Type application/json
http-check expect status 200
server elastic-apm-server-001 10.10.10.11:8080 check port 8080
server elastic-apm-server-002 10.10.10.12:8080 check port 8080
...
I think it's because the header changes as the exporter's request goes through HAProxy.
So I tried setting the Content-Type via the set-header and add-header setting in the HAProxy frontend and backend configuration, but to no avail.
# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
otlphttp:
...
otlphttp configures the OTLP/HTTP exporter. Support for OTLP/HTTP was added to Elastic APM Server in v8.3.0 -- see https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.3.html#_added_7
# /etc/otelcol/config.yaml - working properly
...
exporters:
otlp:
...
otlp configures the OTLP/gRPC exporter.
APM Server v7.12.1 does have support for OTLP/gRPC, but not OTLP/HTTP.