Search code examples
kuberneteskustomizeargocdgitops

Create variable in kustomize manifest

I have what I would consider a common use case but I am really struggling to find a solution:

I want to reuse a variable in Kustomize patches in our deployments. Specifically, we are using commit IDs to reference image tags (Use Case A) and k8s Jobs related to the deployments (Use Case B).

We use a setup where for each ArgoCD app we have a /base/ folder and /overlays/[environment-name], this base is patched with a kustomization.yaml.

Use Case A:

A very straightforward usage - in /overlays/[environment-name] we have a kustomization.yaml which uses:

images:
- name: our-aws-repo-url
  newName: our-aws-repo-url
  newTag: commit-id

Works like a charm since we can re-use this both for the Deployment itself as well as its related Jobs all with one commit reference.

Use Case B:

The problem:

We use N Jobs to e.g. do migrations for 0 downtime deployments where we run alembic containers that run the migration and we have a waitforit initContainer that listens for the Job to get completed i.e. when the migration was successful in order to deploy.

The problem is now that I need to touch 4 files in one service's overlay to patch the id everywhere (which we use to recognize the Job):

  • deployment.yaml like so:
- image: groundnuty/k8s-wait-for:v1.4
  imagePullPolicy: IfNotPresent
  args: 
   - "job"
   - "job-commit-id"
  • job.yaml itself to change re-trigger of Job for new deployment/potential migration:
apiVersion: batch/v1
kind: Job
metadata:
  name: job-commit-id
  • kustomization.yaml as described in Use Case A.

What I think should be possible instead is to:

  1. define variable commit-id somehow in kustomization.yaml and
  2. for Use Case A & B do something like:
apiVersion: batch/v1
kind: Job
metadata:
  name: job-${commit-id}


- image: groundnuty/k8s-wait-for:v1.4
  imagePullPolicy: IfNotPresent
  args: 
   - "job"
   - "job-${commit-id}"

images:
- name: our-aws-repo-url
  newName: our-aws-repo-url
  newTag: ${commit-id}

Goal: when developers do PRs for releases, they should only touch one reference to the commit ID to prevent typos etc. (also easier to review instead of checking commit ID in N places)

Caveat: I am sure there is also another way to do migrations instead of Jobs but this is generally a common thing: how to re-use a variable inside kustomize.

I know I can reference ENV variable in kustomize but I want to reuse a variable within the manifests.

Solution

  • but I want to reuse a variable within the manifests.

    This is not how you typically work with Kustomize. Its a good thing that things are declarative and explicit when working with Kustomize.

    when developers do PRs for releases, they should only touch one reference to the commit ID to prevent typos etc. (also easier to review instead of checking commit ID in N places)

    yes and no.

    That there is a change in four places should not be seen as a problem, in my opinion. That there is human toil to update four places is the problem.

    The solution to human toil is typically automation. By using yq in an automated pipeline (e.g. Jenkins - or a shellscript) can you automate your manifest-update to take a single parameter - this can optionally be automated directly for each build after you have a git "commit id" available. The pipeline need to run four yq-commands to update the four Yaml fields. See e.g. assign operation and github action - pipeline example. No other variables are needed.