dbt: Get return of callback function

I am trying out using dbt programmatically, in particular callbacks. I would like to get the data from the StatsLine event:

from dbt.cli.main import dbtRunner, dbtRunnerResult
from dbt.events.base_types import EventMsg

cli_args = ['run', '-m', 'my_model']

def callback_get_stats(event: EventMsg):
  if event.info.name == 'StatsLine':
    print(event.data)

dbt = dbtRunner(callbacks=[callback_get_stats])
res = dbt.invoke(cli_args)

So far, so good: This prints the data part of the StatsLine event.

Question: How can I get this as an object?

One option would be to use side-effects, e.g.:

callback_dict = dict()

def callback_get_stats(event: EventMsg):
  if event.info.name == 'StatsLine':
    print(event.data)
    callback_dict['StatsLine'] = event.data

This is fine, but it doesn't seem very elegant. I can of course add a return to the function...

def callback_get_stats(event: EventMsg):
  if event.info.name == 'StatsLine':
    print(event.data)
    return event.data

... but I don't know where it goes! I have tried dir() on res and its subobjects but I can't find the return value of callback_get_stats.

This question has a similar flavour. I am using version 1.7.7 of dbt-core.

Solution

Very interesting use case. The scenario actually is quite common. It does not really bind too much with dbt, even not too much about python. It can be for example applied for javascript as well. It is mostly related to the concept of callback.

A callback is designed to do something once an event happened. In this particular case when dbtRunner meet the criteria of EventMsg, it will call all the registered callbacks.

The main confusion comes to the return value. The return value is provided by a callback, to the framework of dbtRunner. The return value is NOT used by the callback itself at all.

If the real action can not be finished inside the callback itself. Then it has to use some external variable, or some event system.

The solution you mention with side effect is a quite common one.