Nomad Variables are not accessible in Job specification

Question

I am new to Nomad and trying to figure out how can I make a use of variables in Nomad using template and I am struggling to figure out how that can be achieved.

Here is my partial job specification which uses template write variable in a ${NOMAD_SECRETS_DIR} and access it as environment variable

    template {
            destination = "${NOMAD_SECRETS_DIR}/env.vars"
            env         = true
            data        = <<EOH
    {{ range nomadVarList "nomad/jobs@arch-team" }}
      {{ . }}
    {{ end }}
    EOH
         }

Where arch-team is the namespace in which variables are created as seen below

Here are my two variables:

Here is my job specification where I am trying to access those variables (role_id and temp_environment) in env stanza.

job "email-api" {
  datacenters = ["dc1"]
  node_pool = "rhel-8x"
  namespace = "arch-team"

  group "api" {
     scaling {
      min     = 1
      max     = 10
      enabled = true
    }

    update {
        max_parallel      = 3
        health_check      = "checks"
        min_healthy_time  = "10s"
        healthy_deadline  = "5m"
        progress_deadline = "10m"
        auto_revert       = true
        auto_promote      = true
        canary            = 1
        stagger           = "30s"
    }

    spread {
    attribute = "${node.unique.id}"
    }

    network {
        port  "http"{
            to = 80
        }
    }

    service {
        name = "emailhandler-api"
        port = "http"
    }

    task "server" {
        template {
            destination = "${NOMAD_SECRETS_DIR}/env.vars"
            env         = true
            data        = <<EOH
                {{ range nomadVarList "nomad/jobs@arch-team" }}
                {{ . }}
                {{ end }}
                EOH
         }

         env {
            environment    = "${NOMAD_PORT_http}"
            NODE_IP = "${NOMAD_IP_http}"
            ASPNETCORE_ENVIRONMENT = "${temp_environment}"
            HashiVaultRoleId_arch = "${role_id}"
          }
          driver = "docker"
          config {
            image = "myregistry/sr.emailhandler.webapi:0.1.22"
            ports = ["http"]
            force_pull = true
          }
        }
    }
}

I exported $NOMAD_TOKEN which is really a management token… When I plan and run job, it just waits and then fails.

I just don’t know how to check what might have happened while trying to get those variables. Is there anything I need to do to read these variables? As I mentioned, I am running job using my nomad_token which is a management token.

Also if someone can tell me how to access the detailed logs of a job I submitted. I tried nomad alloc logs <allocation_id> but it only prints logs that my job writes on console. I don’t see any logs from Nomad.

I will appreciate help in this regards. I am struggling from last couple of days with this.

EDIT 1

Based on suggestion from @KamilCuk, I created workload ACL Policy that looks something like this (hcl file called jobs-variables-policy.hcl):

namespace "arch-team" {
  variables {
    path "*" {
      capabilities = ["read"]
    }
  }
}

I then applied policy as follows:

nomad acl policy apply -namespace arch-team -job email-api email-api-policy ./jobs-variables-policy.hcl

I also made changed to a template which now looks something like this:

template {
        destination = "${NOMAD_SECRETS_DIR}/envs.txt"
        env         = true
        data        = <<EOH
{{ range nomadVarList "nomad/jobs" }}{{ . }}{{ end }}
EOH
     }

Now when I run the job, I get the following error while allocating:

Killing: Template failed to read environment variables: error parsing env template "/opt/nomad/data/alloc/fe60924e-a929-39e6-7c9d-8c3c83ec2621/server/secrets/envs.txt": error on line 1: missing =

EDIT 2

Little more progress after suggestions from @KamilCuk I made changes to a template as seen below:

template {
        destination = "${NOMAD_SECRETS_DIR}/envs.txt"
        env         = true
        data        = <<EOH
{{with nomadVar "nomad/jobs@arch-team" }}
    {{range $k, $v := . }}
      {{ $k }}={{ $v }}
{{ end }}
{{ end }}
EOH
     }

Looks like it is getting those key/value pair.. but there is a lot of whitespace around it as seen below:

Also When I refer to ${environment} in my env stanza in my task, it doesn't have that value of dev

Answer 1

Typically I use nomad variables like the following:

{{with nomadVar "nomad/jobs/<the_name_of_the_job>"}}
ASPNETCORE_ENVIRONMENT={{.environment}}
{{end}}

The {{.}} does not egenerate k=v format readable by env=true, but some internal Go object representation unreadable to anyone.

The nomad/jobs nomad variable directory should be accessible by any job, you can share your variables between all jobs within a namesapce there.