Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying column_types doesn't work on seeds. #458

Open
jausanca opened this issue Oct 7, 2024 · 0 comments · May be fixed by #459
Open

Specifying column_types doesn't work on seeds. #458

jausanca opened this issue Oct 7, 2024 · 0 comments · May be fixed by #459
Labels
bug Something isn't working

Comments

@jausanca
Copy link
Contributor

jausanca commented Oct 7, 2024

Describe the bug

When executing dbt seed, the column types specified on the seed properties are not properly applied, they are set to string instead. Also it gives an error where it says it can infer type if a column is empty, even when specified.

Steps To Reproduce

Execute dbt seed with the column types specified on the .yml properties

Expected behavior

Specified properties should be applied to the columns.

Actual behaviour

String type is applied to columns

System information

The output of dbt --version:

Core:
  - installed: 1.8.7
  - latest:    1.8.7 - Up to date!

Plugins:
  - spark: 1.8.0 - Up to date!

The output of python --version:

Python 3.10.12

Additional context

Looks like the issue is that when a column_type is specified in core is casted as a string so the actual casting is handled by the adapter. Also when the csv data is passed via the executed statement, the spark dataframe is created with no schema, causing it to fail if the column is empty, also some datatypes such as dates are lost on this conversion.

@jausanca jausanca added the bug Something isn't working label Oct 7, 2024
@jausanca jausanca linked a pull request Oct 7, 2024 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant