Send manual OTLP/HTTP (opentelemetry) request to Elastic APM via cURL/Postman

Question

I'm trying to manually send an OTLP/HTTP trace event to Elastic APM (in Elastic Cloud) via Postman.

I've managed to find some JSON examples of OTLP trace requests to produce the following cURL request:

curl --location --request POST 'https://REDACTED.apm.ap-southeast-2.aws.found.io/intake/v2/events' \
--header 'Content-Type: application/json' \
--header 'kbn-xsrf: true' \
--header 'Authorization: ApiKey REDACTED' \
--data-raw '{
  "resource_spans": [
    {
      "scope_spans": [
        {
          "spans": [
            {
              "trace_id": "1",
              "span_id": "1",
              "name": "test",
              "start_time_unix_nano": 1544712660300000000,
              "end_time_unix_nano": 1544712660600000000
            }
          ]
        }
      ]
    }
  ]
}'

... but I can't use JSON to the APM endpoint - APM Server supports both the (OTLP/gRPC) and (OTLP/HTTP) protocol with ProtoBuf payload. APM Server does not yet support JSON Encoding for OTLP/HTTP.

How do I convert the JSON message to ProtoBuf and what would be the Content-Type?

Answer 1

Content-Type will be application/x-protobuf. But payload will be binary, so curl is not the best tool for that.

There are more suitable tools, e.g. protocurl (actually, it only prepares data in the right format for curl, protocurl repo). protocurl command may looks like:

PAYLOAD=$(< trace.json)

protocurl \
 -i opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest \
 -o opentelemetry.proto.collector.trace.v1.ExportTraceServiceResponse \
 -u http://<domain>:<port>/v1/traces \
 -d "$PAYLOAD" -I . -v -D

This generates request (please note request binary - that is actual payload):

...
=========================== Request JSON     =========================== >>>
{"resource_spans":[{"resource":{"attributes":[{"key":"service.name", "value":{"string_value":"demo_lambda_x"}}, {"key":"telemetry.sdk.name", "value":{"string_value":"opentelemetry"}}, {"key":"telemetry.sdk.version", "value":{"string_value":"0.0.1"}}]}}]}
=========================== Request Binary   =========================== >>>
00000000  0a 6c 0a 6a 0a 1f 0a 0c  73 65 72 76 69 63 65 2e  |.l.j....service.|
00000010  6e 61 6d 65 12 0f 0a 0d  64 65 6d 6f 5f 6c 61 6d  |name....demo_lam|
00000020  62 64 61 5f 78 0a 25 0a  12 74 65 6c 65 6d 65 74  |bda_x.%..telemet|
00000030  72 79 2e 73 64 6b 2e 6e  61 6d 65 12 0f 0a 0d 6f  |ry.sdk.name....o|
00000040  70 65 6e 74 65 6c 65 6d  65 74 72 79 0a 20 0a 15  |pentelemetry. ..|
00000050  74 65 6c 65 6d 65 74 72  79 2e 73 64 6b 2e 76 65  |telemetry.sdk.ve|
00000060  72 73 69 6f 6e 12 07 0a  05 30 2e 30 2e 31        |rsion....0.0.1|
Found curl: /usr/local/bin/curl
Invoking curl http request.
Understood additional curl args: []
Total curl args:
  -s
  -X
  POST
  --data-binary
  @/tmp/protocurl-temp-366102455/request.bin
  --output
  /tmp/protocurl-temp-366102455/response.bin
  --dump-header
  /tmp/protocurl-temp-366102455/response-headers.txt
  -H
  Content-Type: application/x-protobuf
...

See manual of protocurl to understand all parameters. Of course you need proto-files - those are "mappings" from human friendly json to machine friendly binary format - https://github.com/open-telemetry/opentelemetry-proto.

A lot of hassles. I would start standard otel collector and configure it properly, e.g. send traces to APM. Then you can send traces in plain text json format with curl to otel collector and collector will do that proto magic before pushing to APM.

BTW: Postman supports GRPC, but again it is more complicated than plain JSON request.