Validation on incremental strategies does not allow for custom strategies

[Ruth-wj]

Describe the bug

Custom strategies are supported in dbt-core and allow users to write macros that define incremental merge startegies ( https://docs.getdbt.com/docs/build/incremental-strategy#custom-strategies ). As long as the strategy macro is found in the namespace, it can be used.

The databricks adapter, however, overrides this functionality. A validation function is used which hardcodes only the builtin incremental strategies, constraining the user unnecessarily. The validation function is here:

dbt-databricks/dbt/include/databricks/macros/materializations/incremental/validate.sql

Line 38 in 6b29d32

{% if raw_strategy not in ['append', 'merge', 'insert_overwrite', 'replace_where'] %}

Personally I have overridden this validation function to include my custom merge strategies, but obviously this isn't ideal.

Steps To Reproduce

Implement custom merge strategy following dbt documentation

Expected behavior

Custom merge strategy should be usable provided they are in the namespace

[benc-db]

I'm fairly sure we imported/inherited this behavior from dbt-spark. At this point I have no idea why the limitation exists, so I will discuss with my team about lifting it.

[mcowart123]

Same issue: 753

[mcowart123]

Has there been any discussion had/movement on this?

[benc-db]

I'm in favor of removing the limitation, but haven't had time to comb the code to understand impact. There may be a couple of places where we assume a fixed set of strategies and make some assertions based on that.

[ldnicolasmay]

Same thing happens with BigQuery adapter:

19:44:52  Finished running 1 view model, 51 data tests, 2 incremental models in 0 hours 0 minutes and 8.93 seconds (8.93s).
19:44:52
19:44:52  Completed with 1 error and 0 warnings:
19:44:52
19:44:52    Compilation Error in model int_my_model (models/intermediate/int_my_model.sql)
  Invalid incremental strategy provided: insert_only
      Expected one of: 'merge', 'insert_overwrite'

  > in macro dbt_bigquery_validate_get_incremental_strategy (macros/materializations/incremental.sql)
  > called by macro materialization_incremental_bigquery (macros/materializations/incremental.sql)
  > called by model int_my_model (models/intermediate/int_my_model.sql)
19:44:52
19:44:52  Done. PASS=18 WARN=0 ERROR=1 SKIP=35 TOTAL=54