Replies: 1 comment
-
I agree with your idea of importing BigQuery descriptions into dbt YAML files! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Description:
I am building a data pipeline from large source raw tables, with hundreds of columns.
I describe my sources in a yaml file in dbt using the package codgen.
The problem with that package is that it only generates empty description placeholders but not the actual tables and columns descriptions.
That is normal because the description is not retrieved by dbt it is not an attribute of the BigqueryColumn class.
My idea is that when calling the bigquery API here with the function get_table from bigquery client API. We retrieve the description from the returned table object such as the table description and the columns description. This will make the column and table description available in the Relation class to be used by the codgen package.
Why is this feature important ?
When dealing with large source data such as tables from SAP, copying manually hundreds of descriptions can be tedious, using codgen package helps but it doesn't generate descriptions.
What do you think about it ?
Beta Was this translation helpful? Give feedback.
All reactions