I am using dbt-core on premises, I own multiple databases with I will apply a transformation process on them using DBT. I also have an Airflow running on my cluster which I am also willing to use to schedule DBT models. My problem is serving the DBT lineage, data and documentation for the data analytics team as well as for our organization.
What it tried/thought of so far:
Build a big project including multiple databases and models divided to multiple databases and categories.
Try and generate multiple projects(?) which didn’t really work using 'dbt serve'
Import other dbt projects using the 'packages.yml'
I have multiple dbt projects which I would like to serve(dbt docs serve) on the web-server for better documentation and visualization(lineage) for my data analysts team.
The option of having separate dbt projects that you import as packages is in my opinion the best. It enables having separate repository for different teams / database while having a central documentation available and still easy to update.
Here is a post from the dbt team discussing this issue, and its pros and cons.