ElasticSearch and Apm search phase execution exception: [script_exception] Reason: link error

Question

I installed ElasticSrarch and Kibana after that added integrated fleet and Apm on windows server 2019 and 2022 (both active now and all stack version is 8.1.2). for client I use AspNetCore sample form OpenTelemetry Github repository. I haven't any problem When I send Apm to windows server 2022 but when I send to 2019 and try access apm services from observability I get below error.

And this error logged:

[2022-04-18T14:34:22.619+04:30][ERROR][plugins.apm] Error: search_phase_execution_exception: [script_exception] Reason: link error
    at D:\ElasticStack\kibana\x-pack\plugins\observability\common\utils\unwrap_es_response.js:60:11
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at getServiceInstancesSystemMetricStatistics (D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\get_service_instances\get_service_instances_system_metric_statistics.js:126:20)
    at async Promise.all (index 1)
    at D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\get_service_instances\main_statistics.js:28:51
    at async Promise.all (index 0)
    at handler (D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\route.js:622:45)
    at D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\apm_routes\register_apm_server_routes.js:143:13
    at Router.handle (D:\ElasticStack\kibana\src\core\server\http\router\router.js:163:30)
    at handler (D:\ElasticStack\kibana\src\core\server\http\router\router.js:124:50)
    at exports.Manager.execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\toolkit.js:60:28)
    at Object.internals.handler (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\handler.js:46:20)
    at exports.execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\handler.js:31:20)
    at Request._lifecycle (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\request.js:371:32)
    at Request._execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\request.js:281:9)
Response: {
  error: {
    root_cause: [
      {
        type: 'script_exception',
        reason: 'link error',
        script_stack: [ "doc['system.memory.actual.free']", '     ^---- HERE' ],
        script: "1 - doc['system.memory.actual.free'] / doc['system.memory.total']",
        lang: 'expression'
      }
    ],
    type: 'search_phase_execution_exception',
    reason: 'all shards failed',
    phase: 'query',
    grouped: true,
    failed_shards: [
      {
        shard: 0,
        index: '.ds-metrics-apm.internal-default-2022.04.17-000001',
        node: 'KsqpLySITbOWAlCPlvsdSg',
        reason: {
          type: 'script_exception',
          reason: 'link error',
          script_stack: [ "doc['system.memory.actual.free']", '     ^---- HERE' ],
          script: "1 - doc['system.memory.actual.free'] / doc['system.memory.total']",
          lang: 'expression',
          caused_by: {
            type: 'parse_exception',
            reason: 'Field [system.memory.actual.free] does not exist in mappings'
          }
        }
      }
    ]
  },
  status: 400
}

I compare both servers and agents configs but I can't find any differences. Why this message and error appears and how can I solve it?

Answer 1

I found the problem two weeks ago, and After some busy days, I will send it here. Also, I think It's a bug.
In a general and fast answer, you should add APM metrics that contain CPU and memory benchmarks. In the below lines, I describe my case in detail.
Windows Server 2022 VPS is a development test environment, and I started my project with the Elastic APM library. This library collects CPU and memory metrics and sends them as APM Metrics by default configurations. After that, I changed my library and used OpentTelemetry. The new one does not send CPU and memory benchmarks, and I ignore it on tests because it's not noticeable for my usage. When I moved from the development to the staging environment with a clean installation of Elastic, an error appeared. After some debugging, I find that the problem solves if I add CPU and Memory metrics. I added metrics by sending them with my first Elastic APM Library, but others can use Kibana and add some fake metrics. For the second approach, you can see the Ovidiu comment.