GitOps: Roll out changes to k8s resources across multiple namespaces

Question

We have a multi-tenant kubernetes cluster, that hosts various customer environments. Kubernetes namespaces are used to isolate these customer environments. Each of these namespaces have a similar set of k8s resources (deployments, configmap etc) configured. Some of these resources are identical across namespaces, while some are not identical.

From a continuous delivery perspective, I am exploring options to roll out changes to these identical components (across namespaces) in a seamless manner. Git-Ops/Pull-based-Continuous-Delivery seems to be a good approach, which would enable us to seamlessly manage 100s of namespaces across various clusters.

Explored a couple of Git-Ops tools such as ArgoCD, Fluxv2 etc... but couldn't figure out whether these tools would let you roll out changes to multiple namespaces simultaneously (or within a predictable time window). It will be helpful, if you can guide/advice me in terms of choosing the right tool/approach to perform roll out to multiple namespaces. It will also be good to understand whether these Git-Ops tools can be customised to handle such scenarios.

An illustration of how namespaces are setup in our k8s cluster.

Namespace: customer1
Deployments: app1, common-app, common-service
Configmaps: cm1, common-cm

Namespace: customer2
Deployments: app2, common-app, common-service
Configmaps: cm2, common-cm

common-app, common-service, common-cm are identical across environments/namespaces.

app1, cm1, app2, cm2 are not identical. They have different image tags, labels etc.

Answer 1

I'm using Fluxv2. My approach will be to create charts for common-app, common-service, etc with same values and define the namespace on separate HelmReleases

So you can put the common-app chart on clusters/production/charts/common-apps and refer the chart path on multiple HelmReleases which can look like this

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: common-apps-customer1
  namespace: customer1
  labels:
    chart: common-apps
spec:
  interval: 1m
  releaseName: common-apps
  chart:
    spec:
      chart: clusters/production/charts/common-apps
      sourceRef:
        kind: GitRepository
        name: manifests
        namespace: flux-system
  values:
    some_key: some_value
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: common-apps-customer2
  namespace: customer2
  labels:
    chart: common-apps
spec:
  interval: 1m
  releaseName: common-apps
  chart:
    spec:
      chart: clusters/production/charts/common-apps
      sourceRef:
        kind: GitRepository
        name: manifests
        namespace: flux-system
  values:
    some_key: some_value

Update

for ns in $(kubectl get ns --no-headers | cut -d " " -f1); do    
    flux create helmrelease common-apps-$ns \
    --source=GitRepository/manifests \
    --chart=common-apps \
    --release-name=common-apps \
    --target-namespace=$ns \
    --interval=1m0s \
    --export > ./clusters/production/common-apps-$ns.yaml     
done;

This will generate the necessary manifests that should be anyway synced with your repo.