Identifying old fields in yml docs in order to remove them

[thomasaarholt]

We have some dbt models that have undergone several refactors, which has led to some fields in the model/schema's yaml documentation longer present being in the model (the maintainer has missed deleting them).

Is there any way to highlight
a) which docs contain unused fields
b) which models have fields that are missing documentation?

As a tiny example for a), I might have the following dbt model:

# cool_model.sql
with my_big_model as (
    select * from {{ ref('my_big_model') }}
)

select
    foo
from my_big_model

and this accompanying yaml file, where bar is now outdated and should be removed, but we haven't identified it yet:

version: 2

models:

  - name: cool_model
    description: 'contains awesome stuff'
    columns:
      - name: foo
        description: 'the foo field'
      - name: bar
        description: 'the bar field'

[jtcohen6]

Hey @thomasaarholt! There's a relatively new feature in dbt-core that enables you to explicitly define the names and data types of columns that will be produced by that model's SQL definition. This isn't exactly what you're asking, but you could set an enforced contract on your model(s), and dbt will return an error message with the diffs between column names/types in your SQL definition and yaml contract:

• https://docs.getdbt.com/docs/collaborate/govern/model-contracts
• https://docs.getdbt.com/reference/resource-configs/contract#example

models:
  - name: cool_model
    description: 'contains awesome stuff'
    config:
      contract: {enforced: true}
    columns:
      - name: foo
        data_type: text
        description: 'the foo field'
      - name: bar
        data_type: text
        description: 'the bar field'

$ dbt run -s cool_model
...
Compilation Error in model cool_model (models/cool_model.sql)
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.

  | column_name | definition_type | contract_type | mismatch_reason       |
  | ----------- | --------------- | ------------- | --------------------- |
  | bar         |                 | TEXT          | missing in definition |

[thomasaarholt]

Ooooh, this is very cool! But I agree, not quite what I wanted here, since this is more about removing old documentation references, keeping our model documentation lean.

It would be great to be able to "set the terms" of the contract, e.g. we must require:

• type
• description
• (maybe something else? can't really think of anything right now)

[dbeatty10]

@thomasaarholt I haven't used it myself, but you could try out the dbt_meta_testing package.

The documentation for that package describes some similar features here:
https://github.com/tnightengale/dbt-meta-testing#required-docs