I have an Airflow environment (v2.4.3) on Kubernetes and I want to sync it with a private git repo so that any changes I make to DAGs in my master branch get automatically picked up by my Airflow environment.
According to Airflow documentation, I can use Git-sync sidecar along with an SSH key added to my private git repo and Airflow env to make it work.
However, given that I am constantly creating new private repos and Airflow environments, I am wondering if there is a more simple way of connecting my private git repos to their respective Airflow environment.
If I have a webapp managing my Airflow environments and have access to an OAuth token from Github after signing into my account (or any other git service), could I use that to connect my an Airflow environement and sync changes to any git repo of my choice under my account?
I was able to figure it out.
One can use personal access tokens as passwords provided by whatever git service the private repo is in along with the repo's username.
I just stored the personal access token as an Opaque secret in my Airflow K8s cluster and referenced that in my git-sync sidecar container yaml definition which I included in my Airflow yaml deployment definition.
containers:
- name: git-sync
image: registry.k8s.io/git-sync/git-sync:v3.6.5
args:
- "-wait=60"
- "-repo=<repo>"
- "-branch=master"
- "-root=/opt/airflow/dags"
- "-username=<username>"
- "-password-file=/etc/git-secret/token"
volumeMounts:
- name: git-secret
mountPath: /etc/git-secret
readOnly: true
- name: dags-data
mountPath: /opt/airflow/dags
volumes:
- name: dags-data
emptyDir: {}
- name: git-secret
secret:
secretName: github-token