Configuring prometheus to access another cluster that applications deployed

Question

I am newbie for using monitoring tools such as Prometheus in k8s..We have two separate cluster one for applications we deployed one for we only would like deploy monitoring,logging tools.

But I have some confusion how to handle this?

1.How cluster that serves prometheus can connect to application cluster and able to pull metrics?

2.How should I specify the namespace if I would like to set a network policy?

3.What should I do in application side for helm chart except exporting metrics?

# Allow traffic from pods with label app=prometheus in namespace with label name=monitoring
# to any pod in <YOUR_APPLICATION_NAMESPACE>
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: monitoring.prometheus.all
  namespace: <YOUR_APPLICATION_NAMESPACE>
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
      podSelector:
        matchLabels:
          app: prometheus
  podSelector: {}
  policyTypes:
  - Ingress

Answer 1

Isn't that what you want?

1) Prometheus federation

Prometheus federation is a Prometheus server that can scrape data from other Prometheus servers. It supports hierarchical federation, which in our case resembles a tree.

A default version of the Prometheus server is installed in each one of our clusters and a Prometheus federation server is deployed together with Grafana in a central monitoring cluster. Prometheus federation scrapes data from all the other Prometheus servers that run in our clusters. For future expansion, a central Prometheus federation can be used to scrape data from multiple Prometheus federation servers that scrape data from groups of tens of clusters.

More info here: https://developers.mattermost.com/blog/cloud-monitoring/

2) Prometheus configuration to scrape Kubernetes outside the cluster yaml example

3) Linkedin Monitoring Kubernetes with Prometheus - outside the cluster! article and Reddit Monitoring K8s by Prometheus Outside Cluster related discussion