When running a native-image of a Micronaut application retrieving Prometheus metrics fails because it cannot get jvm metrics.
Of course native-image pre-compiles the application and includes only what's actually being used, but what kind of JVM is used when running a native-image? I assume it's also a greatly reduced version of the standard hotspot vm.
So is it even possible to get jvm metrics when running a native image or should other means be used to retrieve cpu and memory stats of the running application?
In this case I used Micronaut but I recon it is common for getting metrics from any native-image application with prometheus metrics.
To reproduce:
- install Micronaut cli (e.g. via Homebrew on Mac)
- install GraalVM 19.2.0 (e.g. via Sdkman on Mac)
- install GraalVM native-image tool: gu install native-image
- create application: mn create-app --features=management,micrometer-prometheus,graal-native-image --build maven metrics-graalvm-image
- update src/main/resources/application.yml to allow anonymous access to prometheus endpoint:
endpoints:
all:
enabled: true
sensitive: false
- run application: man exec:exec
- check prometheus metrics can be retrieved: open http://localhost:8080/prometheus
- build application: mvn clean verify
- create native image: native-image --no-server -jar target/metrics-graalvm-image-0.1.jar
- run native image: ./metrics-graalvm-image
- get prometheus metrics: open http://localhost:8080/prometheus
No with the native image, this exception is thrown:
11:06:51.186 [pool-2-thread-4] ERROR i.m.h.s.netty.RoutingInBoundHandler - Unexpected error occurred: ThreadMXBean methods
com.oracle.svm.core.jdk.UnsupportedFeatureError: ThreadMXBean methods
at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:102)
at com.oracle.svm.core.jdk.SubstrateThreadMXBean.getAllThreadIds(ManagementSupport.java:448)
at io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics.getThreadStateCount(JvmThreadMetrics.java:85)
at io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics.lambda$bindTo$0(JvmThreadMetrics.java:75)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:40)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$3(PrometheusMeterRegistry.java:245)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:69)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:95)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:80)
at io.micronaut.configuration.metrics.micrometer.prometheus.management.PrometheusEndpoint.scrape(PrometheusEndpoint.java:52)
at io.micronaut.configuration.metrics.micrometer.prometheus.management.$PrometheusEndpointDefinition$$exec1.invokeInternal(Unknown Source)
at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:144)
at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2792)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:235)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$19(RoutingInBoundHandler.java:1408)
at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14868)
at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:122)
at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
The GraalVM native-image in the standalone executables uses SubstrateVM - a partial VM.
You can find more details about it on its official GitHub repo here.
Additionally according to the SubstrateVM limitations document, the standard management and debugging interfaces that Java offers are not supported. This is because they require access to Java bytecode, which is not longer available at runtime, in case of the native image.