I have troubles to enable metrics on my GKE after customized fluentd in another namespace.
I add some changes to the fluentd configmap, since GKE default fluentd & configmap in kube-system namespace can't change(changes always get reverted), I deployed the fluentd and event-exporter in another namespace.
But the metrics are missing after I made the change. All the logs are OK, still in the logging viewer.
What needs to be done so GKE can collect the metrics again? Or maybe I'm wrong, is there any way to modify the default fluentd configmap in the kube-system?
I wasn't able to find anything useful on this topic. So I create a GCP support ticket. Google provided one solution:
With Cloud Operations for GKE, you can collect just system logs [1] that way monitoring remains enabled in your cluster. Please note that this option can be enabled only via console but not via gcloud command line. There is a tracking bug, https://issuetracker.google.com/163356799 for the same.
Further, you can deploy your own configurable Fluentd daemonset to customize the applications logs [2]
You will be running 2 daemonsets for fluentd with this config, however to reduce the amount of log duplication it would be recommended that you decrease the logging from CloudOps to capture system logs only[2], while your customized fluentd daemonset will be able to capture your application workload logs.
The disadvantages from using this approach are: ensuring your custom deployment doesn't overlap something CloudOps is watching (ie. files, logs), there will be an increased amount of API calls and you will be responsible for updating/maintaining and managing your custom fluentd deployment.
[2]. https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd