We are planning to implement OTEL with Kubernetes and I can't seem to find the correct answer when it comes to k8s.
The question is, do I need/should have the collector agent on each node or just have an Operator for the whole cluster.
What is the best method for OTEL and K8s?
You could deploy a collector as a DaemonSet, which could allow you to not only collect metrics from pods, but also potentially the kubelet and other node-local processes, depending on what metrics you want to gather.
Otherwise, if you have other remote monitoring solutions, and they can read/accept metrics from a collector, then you can deploy a sidecar container into specific namespaces that you want to monitor.