Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create tmp tables with generated unique name. #289

Open
binhnq94 opened this issue Nov 27, 2023 · 0 comments
Open

Create tmp tables with generated unique name. #289

binhnq94 opened this issue Nov 27, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@binhnq94
Copy link

binhnq94 commented Nov 27, 2023

Describe the feature

A clear and concise description of what you want to happen.
With the following configuration:

{{
    config(
        materialized = "incremental",
        file_format = "parquet",
        partition_by = ["created_date"],
        incremental_strategy = "insert_overwrite",
    )
}}

I expected that we could run dbt run -s my_model in parallel with multiple different created_date values passed to vars, for instance, --vars "created_date: 2023-01-01", --vars "created_date: 2023-01-02", ...
However, with the current version 1.7.0, although it seems impossible, dbt-glue will raise an error:

AnalysisException: Table or view 'my_table_tmp' not found in database 'my_db'

For example, if we first run dbt run -s my_model --vars "created_date: 2023-01-01", dbt-glue will create my_model_tmp. Then, if we subsequently run dbt run -s my_model --vars "created_date: 2023-01-02", dbt-glue will attempt to drop my_model_tmp first. This leads to the first run failing as described above.

Describe alternatives you've considered

For each run, need to create tmp table with unique name.

Additional context

Please include any other relevant context here.

Who will this benefit?

Help run incremental model with insert_overwrite strategy in parrallel.

Are you interested in contributing this feature?

I am new to dbt-glue, so I hope for a mentor to guide me through it.

@binhnq94 binhnq94 added the enhancement New feature or request label Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant