v1.9.1

What's Changed

• Remove library restrictions where possible (pandas, pydantic) by @benc-db in #874
• Extend DatabricksRelation types to help codegen by @benc-db in #875
• MERGE EXTENSION: when not matched by source - update by @mi-volodin in #866
• Fix failing concurrency test by @benc-db in #877
• More cleanup work for forward compatibility by @benc-db in #865

Full Changelog: v1.9.0...v1.9.1

v1.9.0

What's Changed

Features

• Add config for generating unique tmp table names for enabling parralel merge (thanks @huangxingyi-git!) (854)
• Add support for serverless job clusters on python models (706)
• Add 'user_folder_for_python' behavior to switch writing python model notebooks to the user's folder (835)
• Merge capabilities are extended (739) to include the support for the following features (thanks @mi-volodin):
  • with schema evolution clause (requires Databricks Runtime 15.2 or above);
  • when not matched by source clause, only for delete action
  • matched, not matched and not matched by source condition clauses;
  • custom aliases for source and target tables can be specified and used in condition clauses;
  • matched and not matched steps can now be skipped;
• Allow for the use of custom constraints, using the custom constraint type with an expression as the constraint (thanks @roydobbe). (792)
• Add "use_info_schema_for_columns" behavior flag to turn on use of information_schema to get column info where possible. This may have more latency but will not truncate complex data types the way that 'describe' can. (808)
• Add support for table_format: iceberg. This uses UniForm under the hood to provide iceberg compatibility for tables or incrementals. (815)
• Add include_full_name_in_path config boolean for external locations. This writes tables to {location_root}/{catalog}/{schema}/{table} (823)
• Add a new workflow_job submission method for python, which creates a long-lived Databricks Workflow instead of a one-time run (thanks @kdazzle!) (762)
• Allow for additional options to be passed to the Databricks Job API when using other python submission methods. For example, enable email_notifications (thanks @kdazzle!) (762)
• Support microbatch incremental strategy using replace_where (825)

Fixes

• Replace array indexing with 'get' in split_part so as not to raise exception when indexing beyond bounds (839)
• Ensure columns that are added get backticked (859)

Under the Hood

• Significant refactoring and increased testing of python_submissions (830)
• Fix places where we were not properly closing cursors, and other test warnings (713)
• Drop support for Python 3.8 (713)
• Upgrade databricks-sql-connector dependency to 3.5.0 (833)
• Prepare for python typing deprecations (837)
• Fix behavior flag use in init of DatabricksAdapter (thanks @VersusFacit!) (836)
• Restrict pydantic to V1 per dbt Labs' request (843)
• Switching to Ruff for formatting and linting (847)
• Refactoring location of DLT polling code (849)
• Switching to Hatch and pyproject.toml for project config (853)

New Contributors

• @huangxingyi-git made their first contribution in #811
• @jelmerk made their first contribution in #820
• @VersusFacit made their first contribution in #836

Full Changelog: v1.8.6...v1.9.0

v1.9.0b1

What's Changed

• Add support for Serverless jobs / refactor api usage by @benc-db in #706
• Extend Merge Capabilities by @mi-volodin in #739
• Upgrade PySql to 3.4.0 by @benc-db in #790
• Add custom constraint option by @roydobbe in #792
• Behavior: Get column info from information_schema Part I by @benc-db in #808
• Simple Iceberg support by @benc-db in #815
• Implement python workflow submissions by @kdazzle in #762
• Behavior for external path by @benc-db in #823
• Implement microbatch incremental strategy by @benc-db in #825

New Contributors

• @mi-volodin made their first contribution in #739
• @roydobbe made their first contribution in #792
• @kdazzle made their first contribution in #762

Full Changelog: v1.8.7...v1.9.0b1

v1.8.7

dbt-databricks 1.8.7 (October 10, 2024)

Features

• Add config for generating unique tmp table names for enabling parallel replace-where (thanks @huangxingyi-git!) (811)

Fixes

• Stop setting 'cluster by' to None. If you want to drop liquid clustering, you will need to full-refresh ([806]#806)
• Don't define table properties on snapshot staging views (thanks @jelmerk!) (820)

New Contributors

• @huangxingyi-git made their first contribution in #811
• @jelmerk made their first contribution in #820

Full Changelog: v1.8.6...v1.8.7

v1.8.6

What's Changed

• Fix table comment updates by @henlue in #750
• Update tblproperties on incremental runs by @benc-db in #765
• Change oauth default scope by @eric-wang-1990 in #776
• Fix issue 747: primary key autoname by @elca-anh in #774
• Fix foreign key constraints by by switching from parent to to and parent_columns to to_columns by @benc-db in #789
• Handle external shallow clone type by @benc-db in #795

New Contributors

• @henlue made their first contribution in #750
• @eric-wang-1990 made their first contribution in #776
• @elca-anh made their first contribution in #774

Full Changelog: v1.8.5...v1.8.6

v1.8.5

What's Changed

• #676 pk constraint by @frankivo in #731
• Fix capitalization issue by @benc-db in #742
• When we persist doc, for python lets remember to add a comment by @benc-db in #743
• Backport warning prevention from 1.9 by @benc-db in #744
• Update setup.py to allow for dbt-adapters ~1.3 by @colin-rogers-dbt in #748
• fix: race condition on cluster creation by @jurasan in #751
• Allow auth type to be configured for tests by @benc-db in #754
• Allow configuring port for tests by @benc-db in #755
• Prep for 1.8.5 release by @yunbodeng-db in #757

New Contributors

• @frankivo made their first contribution in #731
• @jurasan made their first contribution in #751
• @yunbodeng-db made their first contribution in #757

Full Changelog: v1.8.4...v1.8.5

v1.7.17

What's Changed

• Backport close cursor code to 1.7 to silence cursor destructor warnings by @benc-db in #746

Full Changelog: v1.7.16...v1.7.17

v1.8.4

What's Changed

• Fix dbt seed command error when seed file is partially defined in the config file by @kass-artur in #724
• Readd external type by @benc-db in #728
• Adding more tblproperties to ignore list for streaming tables by @benc-db in #736

New Contributors

• @kass-artur made their first contribution in #724

Full Changelog: v1.8.3...v1.8.4

v1.8.3

What's Changed

• Fix #712: missing catalog in one metadata gathering query by @benc-db in #714

Full Changelog: v1.8.2...v1.8.3

v1.8.2

What's Changed

• Reverting the decision to remove 'spark.sql.sources.partitionOverwriteMode = DYNAMIC' for insert_overwrite by @benc-db in #688
• Liquid clustering is now responsive to config changes with incremental models by @benc-db in #686
• Cancel running python jobs on Ctrl-C by @benc-db (thanks @gaoshihang for kicking this off!) in #693
• Change over to system catalog for metadata gathering to fix bug with renamed catalogs by @benc-db in #692
• Fix insert_overwrite replacement when changing from a different partition strategy by @benc-db in #697
• Feature/issue640 refresh spa token no longer writes error log by (thanks @thijs-nijhuis!) in #700
• Migrate remaining unit tests to pytest by @benc-db in #701
• Change default redirect_url and scopes of client dbt-databricks by @jackyhu-db in #704
• Fix serialization issue detected by dbt Labs by @benc-db in #708

New Contributors

• @jackyhu-db made their first contribution in #704

Full Changelog: v1.8.1...v1.8.2