Couldn't see in skywalking ui data from istio metrics.
Using below guide to installed istio and skywalking backend to Kubernetes:
https://github.com/apache/skywalking/tree/master/docs/en/setup/istio.
Made all steps in the guide and deployed app. Deployed app is example book app from istio. Could see traces in Jaeger and works good, but can't see istio metrics in skywalking ui.
How I can provide metrics from istio to skywalking ui or what I need to configure to make metrics from istio come to skywalking ui.
Mixer logs:
gc 1847 @19617.610s 0%: 0.043+22+0.035 ms clock, 0.17+1.9/28/17+0.14 ms cpu, 16->16->8 MB, 17 MB goal, 6 P
2019-10-11T01:28:44.153821Z warn grpcAdapter unable to connect to:/metric.HandleMetricService/HandleMetric, oap.skywalking.svc.cluster.local:11800
{"level":"info","time":"2019-10-11T01:28:43.143390Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"3 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"6.5979ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":1317,"referer":"","reporter":"destination","requestId":"b45592ff-0c49-49fd-af64-3c9229b60466","requestSize":931,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.149845Z","sentBytes":421,"sourceApp":"elasticsearch","sourceIp":"10.1.3.36","sourceName":"skywalking-elasticsearch-data-1","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-data","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-data","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.36"}
{"level":"info","time":"2019-10-11T01:28:43.913917Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"1 error occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"13.906ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":698,"referer":"","reporter":"destination","requestId":"ff7ae8af-f279-46b3-ae86-bde2b61477dd","requestSize":312,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.927780Z","sentBytes":247,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
{"level":"info","time":"2019-10-11T01:28:43.913675Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"8 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"29.8101ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":2430,"referer":"","reporter":"destination","requestId":"b0eec606-c91e-4193-aba2-39b069b76474","requestSize":2044,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.943383Z","sentBytes":857,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
2019-10-11T01:28:44.154193Z error api Report failed: 1 error occurred:
* rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host"
Istio policy:
GC forced
gc 181 @20366.930s 0%: 3.9+19+0.030 ms clock, 15+0/30/67+0.12 ms cpu, 13->13->7 MB, 15 MB goal, 6 P
scvg135: 0 MB released
scvg135: inuse: 15, idle: 43, sys: 58, released: 3, consumed: 55 (MB)
2019-10-11T01:42:28.493381Z info base.baseBalancer: handle SubConn state change: 0xc0011280a0, CONNECTING
2019-10-11T01:42:28.493442Z info roundrobinPicker: newPicker called with readySCs: map[]
2019-10-11T01:42:28.577142Z info grpc: addrConn.createTransport failed to connect to {oap.skywalking.svc.cluster.local:11800 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host". Reconnecting...
2019-10-11T01:42:28.577256Z info base.baseBalancer: handle SubConn state change: 0xc0011280a0, TRANSIENT_FAILURE
GC forced
gc 182 @20486.971s 0%: 0.13+13+0.056 ms clock, 0.55+0/13/26+0.22 ms cpu, 13->13->7 MB, 15 MB goal, 6 P
somehow I have failed to connect to my skywalking backend, problem was with name and namespace but still getting error but now skywalking backend getting data but ui no updates. Istio telemetry:
rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure
Explanation how to set up skywalking properly:
https://github.com/apache/skywalking/issues/3589#issuecomment-543268029