Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting ADBC "numeric" data #20385

Open
diviyank opened this issue Dec 20, 2024 · 0 comments
Open

Supporting ADBC "numeric" data #20385

diviyank opened this issue Dec 20, 2024 · 0 comments
Labels
A-io-database Area: reading/writing to databases enhancement New feature or an improvement of an existing feature

Comments

@diviyank
Copy link

Description

Hello everyone,

As of today, the ADBC driver for PostgreSQL will output all PostgreSQL NUMERIC columns as Arrow strings (see apache/arrow-adbc#767)

And polars does not support that type yet:

line 1213, in arrow_to_pydf
    pydf = PyDataFrame.from_arrow_record_batches(tbl.to_batches())
           │           │                         │   └ <cyfunction Table.to_batches at 0x7f922ffc35e0>
           │           │                         └ pyarrow.Table
           │           │                           assortment_id: int64
           │           │                           sale_start_date: timestamp[us, tz=UTC]
           │           │                           sale_end_date: timestamp[us, tz=UTC]
           │           │                           canal_id: int6...
           │           └ <staticmethod(<built-in method from_arrow_record_batches of type object at 0x558e4899bde0>)>
           └ <class 'builtins.PyDataFrame'>

polars.exceptions.ComputeError: cannot create series from Extension("arrow.opaque", Utf8, Some("{\"type_name\":\"numeric\",\"vendor_name\":\"PostgreSQL\"}"))

This occurs when performing operations on BIGINT columns in the database, that immediately produces NUMERIC types columns.

One way to circumvent the issue at the moment is to manually cast the output type of the columns in the query as the desired type : BIGINT or STR or something else.

Would it be possible to enable conversion for this type?

Thank you!

@diviyank diviyank added the enhancement New feature or an improvement of an existing feature label Dec 20, 2024
@alexander-beedie alexander-beedie added the A-io-database Area: reading/writing to databases label Dec 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io-database Area: reading/writing to databases enhancement New feature or an improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants