Search code examples
yamlgo-templatesargo-workflowsargo

Getting a value from a ConfigMap in my Argo Workflow

My Argo Workflow has a template that generates the following Config Map:

{
  "apiVersion": "v1",
  "kind": "ConfigMap",
  "metadata": { "name": "configmap" },
  "data":
    {
      "ELASTIC_SEARCH_URL": "https://staging-aaa.domain.com",
      "EXTENSION_PATH": "/dist",
      "GRAPHQL_GATEWAY_URL": "https://graphql-gateway.stg.domain.com/graphql",
      "bucket_url": "stg-bucket/fie/to/path",
    },
}

I use this value in one of my other templates like this:

...
envFrom:
- configMapRef:
    name: "{{inputs.parameters.configmap}}"

I also want to get a "2-in-1" by getting bucket_url within that output, so I created this template to test if I'm able to print what I want (I can't withParam over the original output, so I added [] around {{steps.create-configmap.outputs.parameters.configmap}}):

- - name: print-with-loop
    template: print-with-loop-tmpl
    arguments:
      parameters:
        - name: data
          value: "{{item}}"
    withParam: "[{{steps.create-configmap.outputs.parameters.configmap}}]"

The output of this template is exactly the Config Map itself:

{apiVersion:v1,data:{ELASTIC_SEARCH_URL:https://staging-aaa.domain.com,EXTENSION_PATH:/dist,GRAPHQL_GATEWAY_URL:https://graphql.stg.domain.com/graphql,bucket_url:stg-bucket/fie/to/path},kind:ConfigMap,metadata:{name:env-configmap}}

I can also print item.data within that Config Map:

{ELASTIC_SEARCH_URL:https://staging-aaa.domain.com,EXTENSION_PATH:/dist,GRAPHQL_GATEWAY_URL:https://graphql.stg.domain.com/graphql,bucket_name:stg-bucket,ext_path:extension/stable/dist-raw.tar.gz,bucket_url:stg-bucket/extension/stable/dist-raw.tar.gz}

However I can't access any data within item.data. If I use item.data.bucket_url or item.data['bucket_url'], it doesn't work and I get errors from Argo.

I tried to manipulate the output using sprig but I wasn't able to find a solution. Basically I'm trying to fetch bucket_url to use in another template within this workflow.

Reproduce the issue yourself

  1. Run your Argo server
  2. Create a Workflow yaml file
  3. Run argo submit with your new workflow file.
  4. That's it :)

I made the smallest template I can that should produce the exact same result. If you run Argo locally like I do, maybe give it a try:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: cluster
        value: "stg"
      - name: version
        value: "stg/stable"

  templates:
    - name: main
      steps:
        - - name: create-configmap
            template: create-configmap-tmpl

        - - name: print-with-loop
            template: print-with-loop-tmpl
            arguments:
              parameters:
                - name: data
                  value: "{{item}}"
            withParam: "[{{steps.create-configmap.outputs.parameters.configmap}}]"

    - name: create-configmap-tmpl
      inputs:
        artifacts:
          - name: python-script
            path: /tmp/script.py
            raw:
              data: |
                import json

                def create_simple_config_map():
                    config = {
                        "ELASTIC_SEARCH_URL": "https://example-domain.com",
                        "GRAPHQL_GATEWAY_URL": "https://graphql.example.com/graphql",
                        "bucket_name": "example-bucket",
                        "ext_path": "example/path/file",
                        "bucket_url": "example-bucket/example/path/file"
                    }
                    
                    return config

                def main():
                    config = create_simple_config_map()

                    configmap = {
                        "apiVersion": "v1",
                        "kind": "ConfigMap",
                        "metadata": {
                            "name": "env-configmap"
                        },
                        "data": config
                    }

                    with open("/tmp/configmap.json", "w") as file:
                        json.dump(configmap, file, indent=4)

                    print(json.dumps([config], indent=4))

                if __name__ == "__main__":
                    main()
      container:
        image: python:3.11
        command: ["python", "/tmp/script.py"]
      outputs:
        parameters:
          - name: configmap
            valueFrom:
              path: /tmp/configmap.json

    - name: print-with-loop-tmpl
      inputs:
        parameters:
          - name: data
      script:
        image: bash
        command: [bash]
        source: |
          echo "{{inputs.parameters.data}}"

The step create-configmap-tmpl generates a valid Config Map, you can also run it locally:

import json

def create_simple_config_map():
    config = {
        "ELASTIC_SEARCH_URL": "https://example-domain.com",
        "GRAPHQL_GATEWAY_URL": "https://graphql.example.com/graphql",
        "bucket_name": "example-bucket",
        "ext_path": "example/path/file",
        "bucket_url": "example-bucket/example/path/file"
    }
    
    return config

def main():
    config = create_simple_config_map()

    configmap = {
        "apiVersion": "v1",
        "kind": "ConfigMap",
        "metadata": {
            "name": "configmap"
        },
        "data": config
    }

    with open("/tmp/configmap.json", "w") as file:
        json.dump(configmap, file, indent=4)

    print(json.dumps([config], indent=4))

if __name__ == "__main__":
    main()

The output of this script is the following:

{
    "apiVersion": "v1",
    "kind": "ConfigMap",
    "metadata": {
        "name": "configmap"
    },
    "data": {
        "ELASTIC_SEARCH_URL": "https://example-domain.com",
        "GRAPHQL_GATEWAY_URL": "https://graphql.example.com/graphql",
        "bucket_name": "example-bucket",
        "ext_path": "example/path/file",
        "bucket_url": "example-bucket/example/path/file"
    }
}

You can now try to play around with the printing:

- - name: print-with-loop
    template: print-with-loop-tmpl
    arguments:
    parameters:
        - name: data
        value: "{{item}}"
    withParam: "[{{steps.create-configmap.outputs.parameters.configmap}}]"
  • If we use item, it prints the entire Config Map
  • If we use item.data, it would also work.

The problem is accessing item.data.bucket_url or item.data['bucket_url']. It won't work. As mentioned, I tried various sprig functions like toJson, lists and dict manipulation, but nothing worked.

Solution

  • I ended up using a different parameter for withParam:

            - - name: print-with-loop
            template: print-with-loop-tmpl
            arguments:
              parameters:
                - name: data
                  value: "{{item.bucket_url}}"
            withParam: "{{steps.create-configmap.outputs.result}}"

    Since I'm printing the json.dump that I'm doing in the script that generates the ConfigMap, I can easily access bucket_url. The above template's output is exactly what I needed.