We have an existing Kubernetes cluster running that is using Istio. I was planning on adding a new Prometheus pod and can find plenty of blogs on how to do it. However, I noticed Istio already has a Prometheus service running in the Istio-System namespace.
My main goal is to get Grafana running with a few basic monitoring dashboards. Should I go ahead and use Istio's Prometheus service? What are the advantages/disadvantages of using Istio's Prometheus service over running my own?
I'd suggest not sharing the existing istio prometheus, it's deployed in the istio-system namespace for a reason. It was deployed by and configured for istio.
If you really want to create a central shared prometheus service, use prometheus-operator and create a prometheus operator for istio. This is still going to be a lot of config effort to reintegrate your istio installation back into this new prometheus instance and is probably only worth it if you plan on scaling the number of clusters running this setup. 2 or 4 Prometheis is a manageable gap. 20 or 40 not so much.