Search code examples
dbt

How to have dbt run with log error FileNotFound to trigger an exit code that isn't 0?

We're running dbt version 0.16.1. We've set up our data pipeline to run in Airflow, and have a library set up to map each dbt model run within it's own bash operator on Airflow.

The dbt run command executed is as follows:

cd /usr/local/airflow/models/[PACKAGE_NAME] && dbt --log-format json run --models [MODEL_NAME]--no-version-check --profiles-dir=/usr/local/airflow/dags/dags-enterprise-model/enterprise_model/include --target=[TARGET] --profile=[PROFILE]

Occasionally (likely when two models are being run at the same time), Airflow will show the following message from within the dbt run command:

INFO - FileNotFoundError: [Errno 2] No such file or directory: 'logs/dbt.log' -> 'logs/dbt.log.1'

This is problematic because the logfiles do not get updated, but the exit code of the task is listed a 0:

Command exited with return code 0

This causes Airflow to mark the task as a success; however, the log wasn't printed successfully.

My questions:

  1. Is there a a way for these errors to be raised as an actual error?
  2. Failing that, is there a way to specific a unique log file?

I'm not sure if this is a gap in my understand, a bug within dbt's logging, or maybe both?

Solution

  • It definitely sounds like this is the result of invoking dbt multiple times simultaneously, while having it write to the same files. It's not a dbt bug because we don't intend for dbt to be invoked simultaneously; a single invocation can handle concurrent model runs via threads. Log collisions are one risk of reimplementing dbt's model DAG as Airflow DAGs.

    Those are both fair questions:

    1. Historically, dbt only used two log levels: debug and info. See the comment on a related issue: dbt#2680. I totally appreciate that Airflow and other orchestration tools have well defined notification behaviors when presented with different log levels. A community member actually just opened a PR to add error-level logging (dbt#2723).

    2. It is possible to set a custom log path for a dbt invocation using the log-path config in dbt_project.yml (docs)