argocd app create in CI pipeline (GitHub Actions, Tekton, ...) throws "PermissionDenied desc = permission denied: applications, create, default/myapp"

Question

From our Tekton pipeline we want to use ArgoCD CLI to do a argocd app create and argocd app sync dynamically based on the app that is build. We created a new user as described in the docs by adding a accounts.tekton: apiKey to the argocd-cm ConfigMap:

kubectl patch configmap argocd-cm -n argocd -p '{"data": {"accounts.tekton": "apiKey"}}'

Then we created a token for the tekton user with:

argocd account generate-token --account tekton

With this token as the password and the username tekton we did the argocd login like

argocd login $(kubectl get service argocd-server -n argocd --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}') --username=tekton --password="$TOKEN";

Now from within our Tekton pipeline (but we guess that would be the same for every other CI, given the usage of a non-admin user) we get the following error if we run argocd app create:

$ argocd app create microservice-api-spring-boot --repo https://gitlab.com/jonashackt/microservice-api-spring-boot-config.git --path deployment --dest-server https://kubernetes.default.svc --dest-namespace default --revision argocd --sync-policy auto
error rpc error: code = PermissionDenied desc = permission denied: applications, create, default/microservice-api-spring-boot, sub: tekton, iat: 2022-02-03T16:36:48Z

Answer 1

The problem is mentioned in Argo's useraccounts docs:

When you create local users, each of those users will need additional RBAC rules set up, otherwise they will fall back to the default policy specified by policy.default field of the argocd-rbac-cm ConfigMap.

But these additional RBAC rules could be setup the simplest using ArgoCD Projects. And with such a AppProject you don't even need to create a user like tekton in the ConfigMap argocd-cm. ArgoCD projects have the ability to define Project roles:

Projects include a feature called roles that enable automated access to a project's applications. These can be used to give a CI pipeline a restricted set of permissions. For example, a CI system may only be able to sync a single app (but not change its source or destination).

There are 2 solutions how to configure the AppProject, role & permissions incl. role token:

1. using argocd CLI
2. using a manifest YAML file

1.) Use argocd CLI to create AppProject, role & permissions incl. role token

So let's get our hands dirty and create a ArgoCD AppProject using the argocd CLI called apps2deploy:

argocd proj create apps2deploy -d https://kubernetes.default.svc,default --src "*"

We create it with the --src "*" as a wildcard for any git repository (as described here).

Now we create a Project role called create-sync via:

argocd proj role create apps2deploy create-sync --description "project role to create and sync apps from a CI/CD pipeline"

You can check the new role has been created with argocd proj role list apps2deploy.

Then we need to create a token for the new Project role create-sync, which can be created via:

argocd proj role create-token apps2deploy create-sync

This token needs to be used for the argocd login command inside our Tekton / CI pipeline. There's also a --token-only parameter for the command, so we can create an environment variable via

ARGOCD_AUTH_TOKEN=$(argocd proj role create-token apps2deploy create-sync --token-only)

The ARGOCD_AUTH_TOKEN will be automatically used by argo login.

Now we need to give permissions to the role, so it will be able to create and sync our application in ArgoCD from within Tekton or any other CI pipeline. As described in the docs we therefore add policies to our roles using the argocd CLI:

argocd proj role add-policy apps2deploy create-sync --action get --permission allow --object "*"
argocd proj role add-policy apps2deploy create-sync --action create --permission allow --object "*"
argocd proj role add-policy apps2deploy create-sync --action sync --permission allow --object "*"
argocd proj role add-policy apps2deploy create-sync --action update --permission allow --object "*"
argocd proj role add-policy apps2deploy create-sync --action delete --permission allow --object "*"

Have a look on the role policies with argocd proj role get apps2deploy create-sync, which should look somehow like this:

$ argocd proj role get apps2deploy create-sync
Role Name:     create-sync
Description:   project role to create and sync apps from a CI/CD pipeline
Policies:
p, proj:apps2deploy:create-sync, projects, get, apps2deploy, allow
p, proj:apps2deploy:create-sync, applications, get, apps2deploy/*, allow
p, proj:apps2deploy:create-sync, applications, create, apps2deploy/*, allow
p, proj:apps2deploy:create-sync, applications, update, apps2deploy/*, allow
p, proj:apps2deploy:create-sync, applications, delete, apps2deploy/*, allow
p, proj:apps2deploy:create-sync, applications, sync, apps2deploy/*, allow
JWT Tokens:
ID          ISSUED-AT                                EXPIRES-AT
1644166189  2022-02-06T17:49:49+01:00 (2 hours ago)  <none>

Finally we should have setup everything to do a successful argocd app create. All we need to do is to add the --project apps2deploy parameter:

argocd app create microservice-api-spring-boot --repo https://gitlab.com/jonashackt/microservice-api-spring-boot-config.git --path deployment --project apps2deploy --dest-server https://kubernetes.default.svc --dest-namespace default --revision argocd --sync-policy auto

2.) Use manifest YAML to create AppProject, role & permissions incl. role token

As all those CLI based steps in solution 1.) are quite many, we could also using a manifest YAML file. Here's an example argocd-appproject-apps2deploy.yml which configures exactly the same as in solution a):

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: apps2deploy
  namespace: argocd
spec:
  destinations:
    - namespace: default
      server: https://kubernetes.default.svc
  sourceRepos:
    - '*'
  roles:
    - description: project role to create and sync apps from a CI/CD pipeline
      name: create-sync
      policies:
      - p, proj:apps2deploy:create-sync, applications, get, apps2deploy/*, allow
      - p, proj:apps2deploy:create-sync, applications, create, apps2deploy/*, allow
      - p, proj:apps2deploy:create-sync, applications, update, apps2deploy/*, allow
      - p, proj:apps2deploy:create-sync, applications, delete, apps2deploy/*, allow
      - p, proj:apps2deploy:create-sync, applications, sync, apps2deploy/*, allow

There are only 2 steps left to be able to do a successful argocd app create from within Tekton (or other CI pipeline). We need to apply the manifest with

kubectl apply -f argocd-appproject-apps2deploy.yml

And we need to need to create a role token, ideally assigning it directly to the ARGOCD_AUTH_TOKEN for the argocd login command (which also needs to be done afterwards):

ARGOCD_AUTH_TOKEN=$(argocd proj role create-token apps2deploy create-sync --token-only)

The same argocd app create command as mentioned in solution 1.) should work now:

argocd app create microservice-api-spring-boot --repo https://gitlab.com/jonashackt/microservice-api-spring-boot-config.git --path deployment --project apps2deploy --dest-server https://kubernetes.default.svc --dest-namespace default --revision argocd --sync-policy auto