Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PK Constraint fails against cloned table (slim CI setup) #884

Open
kmarq opened this issue Dec 18, 2024 · 1 comment
Open

PK Constraint fails against cloned table (slim CI setup) #884

kmarq opened this issue Dec 18, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@kmarq
Copy link

kmarq commented Dec 18, 2024

Describe the bug

Building a model that was created by doing a clone command that has a PK constraint fails with an error that the PK already exists when dbt tries to apply an alter table to create the PK.

Found the issue when doing a slim CI setup in order to support setting PK/FK constraints I am executing:
dbt clone --select @State:modified,state:old
I then execute
dbt build --select state:modified+
This way all parent models for any model downstream of an impacted model are present, this is helpful for being able to have the FK references work, otherwise the referenced table is potentially not present, and currently FK does not honor deferral.

When building a model that was created by the above clone, if that model has a PK defined, the model fails due to the PK already existing.

 Database Error in model model_b (models/model_b.sql)
  [DELTA_CONSTRAINT_ALREADY_EXISTS] Constraint 'model_b_pk' already exists. Please delete the old constraint first.
  Old constraint:
  model_b_pk PRIMARY KEY (`pk_col`)
  compiled code at target/run/databricks_analytics_dbt/models/model_b.sql

Steps To Reproduce

Create a model with a PK defined:

{{ config(materialized = 'table')}}
select 
'a' as pk_col
- name: model_b
  config:
    contract:
      enforced: true
  columns:
  - name: pk_col
    data_type: string
    constraints:
      - type: primary_key
        name: model_b_pk
      - type: not_null

If you directly build this and rebuild it works fine. If you create this model in a different target schema, and then clone it to your current schema when you try to build it will fail with the constraint already exists error.

I've tried without naming the PK and using the hash. On clone Databricks is automatically generating the PK name as
'tablename'_pk
Using the same name does not get dbt to recognize the already existing PK and so it tries to do an alter to apply it which then fails.

Expected behavior

DBT should recognize an existing PK and either remove it and replace with the current model definition, or especially if the name matches alter the existing version rather than attempting to apply it as a new PK.

Screenshots and log output

Cloned table exists:
image

image

Create or replace statement executes:

create
or replace table `model_b` using delta
as
select
  pk_col
from
  (
    select
      'a' as pk_col
  ) as model_subq

Then tries to alter with the PK

alter table
`model_b`
add
  constraint model_b_pk primary key(pk_col)

System information

DBT Cloud using 'Latest Version'

16:16:32 Registered adapter: databricks=1.9.0-post8+5e20eeaef43e671913f995d8079d4ec2b8a1da6d
16:16:32 checksum: 429d12d886761424119078b59c6289cc952f90a4f6bdc27a6fe96500a5717cce, vars: {}, profile: user, target: , version: 2024.12.18+447b71a

Additional context

Add any other context about the problem here.

@kmarq kmarq added the bug Something isn't working label Dec 18, 2024
@benc-db
Copy link
Collaborator

benc-db commented Dec 18, 2024

Thanks for reporting, will investigate as part of redoing constraint enforcement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants