Execute adding column comments/constraints as a single query

[jsingh-cw]

Currently, if I provide column comments or NOT NULL constraints for 100 columns in a model, dbt-databricks adapter ends up running 200 separate queries resulting in significantly long total model run time. Ideally, all comments should be added via single ALTER query and constraints should be added via second ALTER query.

This will benefit model run time.

Please see attached log to see how individual SQL queries added to the overall model runtime. Table creation/data load takes less than 30 seconds but column comments and NOT NULL constraints are added to one column per SQL query and takes ~7 mins to complete.

debug_run_86601.log

[benc-db]

I'm working on this as we speak. There is no bulk alter for column comments, but I'm reshaping materializations to set comments at create time. This means separating create and insert of data (because you can't set comments with 'create table as select' :), but given how much users have complained about comment performance, I expect it will be a net win.

[jeremyyeo]

Fwiw - users who find their say hourly runs doing persist docs on columns taking too long (cause every single run has those alter table ... change column ddls) - can try to move to perhaps a once daily / once weekly run-op that manually adds those comments. Something like https://github.com/jeremyyeo/the-hitchhikers-guide-to-dbt/tree/main/manually-persisting-docs