Search code examples
yamlamazon-redshiftdbt

dbt is not able to find my source defined in sources.yaml

I'm setting up a new dbt project and trying to define a source then use it in a downstream model.

Here is my sources.yml located under models folder

version: 2

sources:
  - name: raw
    schema: my_schema
    database: my_db
    tables:
      - name: my_table
        description: my description
        columns:
          - name: op
            description: desc
          - name: id
            description: id
......

In my model file my_downstream_table.sql, I put SQL:

{{ config(materialized='table', sort='...', dist='...') }}
  SELECT
    ...
  FROM
    {{source('my_schema', 'my_table')}} p 
  ......

When running without referencing on the source model, the SQL works well.

However, with reference to the source model, When I compile my project by running dbt compile, I received error:

6:10:14  Running with dbt=1.5.0
06:10:16  Encountered an error:
Compilation Error
  Model 'model.myproj.my_downstream_table' (models/my_downstream_table.sql) depends on a source named 'my_schema.my_table' which was not found

I tried to place the sources.yml in different locations or name it differently but all failed with same error.

Here is my dbt_project.yml if it helps to figure out what's wrong with my setup.


# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'myproj'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
  myproj:
    +materialized: table
Solution

  • The first argument to the source() macro should be the source name, not the schema name.

    Try:

    {{ source('raw', 'my_table') }}