Parsing a dbt model in Python in order to extract column names

[thomasaarholt]

Hi!
I would like to parse a dbt model in Python, with the sole purpose of extracting the model names. Is there a way I can use the dbt-core library to do this?

Something ala the following dummy code:

from dbt.something import parse
path = "path_to_dbt_model.sql"
parsed_model = parse(path)

print(parsed_model.columns)

[dbeatty10]

If you are using dbt 1.5 or greater, you could utilize programmatic invocations with a Python script like this named runner.py:

# runner.py

from dbt.contracts.graph.manifest import Manifest
from dbt.contracts.results import CatalogArtifact
from dbt.cli.main import dbtRunner, dbtRunnerResult

# initialize
dbt = dbtRunner()

# use 'parse' command to load a Manifest
res: dbtRunnerResult = dbt.invoke(["parse"])
manifest: Manifest = res.result

# use 'docs generate' command to build a catalog
res: dbtRunnerResult = dbt.invoke(["docs", "generate"])
catalog: CatalogArtifact = res.result

for catalog_node in catalog.nodes.values():

    # lookup the catalog node within the manifest
    manifest_node = manifest.nodes[catalog_node.unique_id]

    # use the manifest to get the model path and relation details
    print(f"path: {manifest_node.original_file_path}")
    print(f"relation_name: {manifest_node.relation_name}")
    print(f"database: {manifest_node.database}")
    print(f"schema: {manifest_node.schema}")
    print(f"alias: {manifest_node.alias}")

    # use the catalog to get column details
    print("columns:")

    for column in catalog_node.columns.values():
        print(f"{column.name} ({column.type})")

    print()

Then run it like this:

python runner.py

The above assumes you've recently built all your database tables using dbt run or dbt build. Otherwise, you may be missing any columns that don't yet exist within the database.

This script iterates through the catalog nodes, but I think you could also invert it and loop through the manifest nodes instead. I believe unique_id is the common key either direction.

Here's some descriptions of the object structures that may be helpful:

• catalog
• manifest