Search code examples
google-cloud-platformdataformgoogle-dataform

dataform schema suffix for sources

I am trying to manage production and development environments using the GCP cloud GUI from the same code base. I have created separate release configurations and workflow configurations based off of the same DAG, with the only difference that the production environment data resides in a schema with _prod as the suffix.

I have accordingly included the correct suffix in the release configuration for production:

enter image description here

However, I have discovered that it does not apply to my source data, which uses schema: dataform.projectConfig.defaultSchema in the declaration. Am I configuring the project/release/workflows correctly?

Solution

  • Solution was pretty simple:

    Update the source declaration so that the schema is a concatenation of two variables:

    schema: dataform.projectConfig.defaultSchema + dataform.projectConfig.vars.source_suffix

    Add the following variable to the dataform.json

      "vars" : {
    "source_suffix" : ""
  }

    For production releases, specify the variable override source_suffix to be _prod.