Search code examples
kubernetesdbt

dbt: Env var required but not provided:

Requirement: To read environment variables values from Kubernetes to dbt profiles.yml file

Issue: profiles.yml unable to read Kubernetes secrets values

Error :

Parsing Error
Env var required but not provided: 'profilePassword'

I run dbt command in Airflow which is installed Kubernetes

Tried below

Created secrets in kubernetes

kubectl create secret generic profile-secret --from-literal=profileusername=ABC_SA --from-literal='password=abcd' -n airflow

updated in the yml file as

secret:
  - envName: profileUser
    secretName: "profile-secret"
    secretKey: "profileusername"     
  - envName: profilePassword
    secretName: "profile-secret"
    secretKey: "password"

In the dbt profiles.yml(Also tried without space)

abc_data:
  outputs:
    dev:
      account: abc
      user: DEV_SA
      password: "{{ env_var('profilePassword') }}"
      database: DEV
      role: DEV-ROLE
      schema: DEV_SCHEMA
      threads: 1
      type: snowflake
      warehouse: DEV_WH
  target: dev

When placed hardcoded password values, it runs fine Or is there better way to read secret values

Solution

  • Resolved , issue was the python subprocesses was not reading the env variables