mapping values are not allowed in this context in "<unicode string>"

Question

In my loop, I run a dbt command and save the output to a .yml file. The following command works and generates a schema in my .yml file accurately:

for file in models/l30_mart/*.sql; do
    table=$(basename "$file" .sql)
    dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > test.yml
done

However, in the example above, I am saving the test.yml file in the root directory. When I try to save the file in another path for example models/l30_mart/test.yml like this, it doesn't work:

for file in models/l30_mart/*.sql; do
    table=$(basename "$file" .sql)
    dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/test.yml
done

In this case, when I open the test.ymlfile, I see this:

12:06:42  Running with dbt=1.0.1
12:06:43  Encountered an error:
Compilation Error
  The schema file at models/l30_mart/test.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
  
  https://docs.getdbt.com/docs/schemayml-files

What am I missing out on?

If I try something like this to save different files with the extracted tablename variable as the filename, it also doesn't work:

for file in models/l30_mart/*.sql; do
    table=$(basename "$file" .sql)
    dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" >  models/l30_mart/$table.yml
done

In this case, the files either have this output:

20:39:44  Running with dbt=1.0.1
20:39:45  Encountered an error:
Compilation Error
  The schema file at models/l30_mart/**firsttable.yml** is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
  
  https://docs.getdbt.com/docs/schemayml-files

or this (eg in the secondtablename.yml file):

20:39:48  Running with dbt=1.0.1
20:39:49  Encountered an error:
Parsing Error
  Error reading dbt_4flow: l30_mart/firstablename.yml - Runtime Error
    Syntax error near line 2
    ------------------------------
    1  | 20:39:44  Running with dbt=1.0.1
    2  | 20:39:45  Encountered an error:
    3  | Compilation Error
    4  |   The schema file at models/l30_mart/firsttablename.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
    5  |   
    
    Raw Error:
    ------------------------------
    mapping values are not allowed in this context
      in "<unicode string>", line 2, column 31

Note that the secondtablename.yml mentions the firsttablename.yml.

Answer 1

I don't know dbt but the explanation that seems likely is that dbt for some reason parses all *.yml files in that target directory when you call it. Since the shell opens the pipe to the *.yml file before calling dbt, the file already exists (but initially empty) when dbt is called. Since dbt expects the file to contain a version, you get an error.

To check whether this assessment is correct, write into a temporary file:

for file in models/l30_mart/*.sql; do
    target_file=$(mktemp)
    table=$(basename "$file" .sql)
    dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > $target_file
    mv $target_file models/l30_mart/test.yml
done

(Be aware of mktemp shenanigans if you're using macOS)

Edit: Since dbt seems to be affected by the files existing, you can also try to generate all files and move them into the correct directory afterwards:

target_dir=$(mktemp -d)
for file in models/l30_mart/*.sql; do
    table=$(basename "$file" .sql)
    dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > $target_dir/$table.yml
done
mv $target_dir/*.yml models/l30_mart/
rmdir $target_dir