Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt enforced data_type checks doesn't check complex data types in Databricks #880

Open
zeljkostojkovic opened this issue Dec 17, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@zeljkostojkovic
Copy link

Describe the bug

When I define model like following, and execute dbt run command, command will succeed if metadata field is really of type STRUCT, but underlying keys in the struct and their corresponding data types won't be checked.

models:
  - name: test_model
    config:
      contract:
        enforced: true
        alias_types: false
    columns:
      - name: id
        data_type: STRING

      - name: metadata
        data_type: STRUCT<property_1 STRING, property_2 STRING>

SQL model:

  select
    id,
    CAST(metadata AS STRUCT<property_3 STRING> as metadata
  from {{ source('source', 'source_table') }}

I've noticed this when I tried testing and changing my data_type from STRUCT to MAP. In that case dbt run command retuned this error:

image

definition_type and contract_type are not taking into account values inside complex/nested structures.

Steps To Reproduce

Using model configuration and similar SQL command (that contains STRUCT field) from above, execute dbt run command. Command will succeed no matter what underlying struct configuration is setup in data_type field (works with setting just STRUCT<>).

Expected behavior

I would expect for dbt to fail if contract is enforced, and complex data types like ARRAY, MAP and STRUCT are not fully described, or even better under contract configuration to have check_complex_types boolean field which can decide whether to check nested structures or not.

System information

The output of dbt --version:

dbt Cloud CLI - 0.38.15
dbt-databricks: 1.8.7

The operating system you're using:
macOS 15.1.1

The output of python --version:
Python 3.11.9

@zeljkostojkovic zeljkostojkovic added the bug Something isn't working label Dec 17, 2024
@benc-db
Copy link
Collaborator

benc-db commented Dec 17, 2024

Thanks for reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants