Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostgreSQL XML data type not supported #1330

Open
nj1973 opened this issue Nov 12, 2024 · 1 comment
Open

PostgreSQL XML data type not supported #1330

nj1973 opened this issue Nov 12, 2024 · 1 comment
Labels
priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) type: feature request 'Nice-to-have' improvement, new feature or different behavior or design.

Comments

@nj1973
Copy link
Contributor

nj1973 commented Nov 12, 2024

Test case from a customer validating Oracle vs PostgreSQL.

PostgreSQL:

CREATE TABLE dvt_test.tab_xml
( id numeric(1000,0) NOT NULL PRIMARY KEY
, batch_id numeric(1000,0) NOT NULL
, xml_data xml);

Oracle:

CREATE TABLE dvt_test.tab_xml
( id NUMBER(*,0) NOT NULL PRIMARY KEY
, batch_id NUMBER(*,0) NOT NULL
, xml_data NCLOB);

Schema validation:

$ data-validation -v validate schema --source-conn ora --target-conn pg --tables-list "dvt_test.tab_xml"
...
    cursor.execute(statement, parameters)
psycopg2.errors.SyntaxError: syntax error at or near "dvt_test"
LINE 1: ...EMPORARY VIEW d8c496e55b324f97acba672aa8610c88 AS dvt_test.t...

Column validation:

$ data-validation -v validate column --source-conn ora --target-conn pg --tables-list "dvt_test.tab_xml"
...
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "dvt_test"
LINE 1: ...PORARY VIEW "26d25695f1eb44ea9fbec0b11ac6190c" AS dvt_test.t...
                                                             ^
[SQL: CREATE TEMPORARY VIEW "26d25695f1eb44ea9fbec0b11ac6190c" AS dvt_test.tab_xml]
(Background on this error at: https://sqlalche.me/e/14/f405)

Row validation:

$ data-validation -v validate row --source-conn ora --target-conn pg --tables-list "dvt_test.tab_xml" -pk=id --hash="*"
...
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "dvt_test"
LINE 1: ...PORARY VIEW "52ee1651e62a4392965f1569fe1a5792" AS dvt_test.t...
                                                             ^
[SQL: CREATE TEMPORARY VIEW "52ee1651e62a4392965f1569fe1a5792" AS dvt_test.tab_xml]
(Background on this error at: https://sqlalche.me/e/14/f405)
@nj1973 nj1973 added the type: feature request 'Nice-to-have' improvement, new feature or different behavior or design. label Nov 12, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Nov 12, 2024

It looks like we can prevent the exception with changes like below, these result in a datatype of unknown which matches what Ibis has done in more recent releases (ibis-project/ibis#8610):

--- a/third_party/ibis/ibis_postgres/datatypes.py
+++ b/third_party/ibis/ibis_postgres/datatypes.py
@@ -13,10 +13,23 @@
 # limitations under the License.

 import ibis.expr.datatypes as dt
+from sqlalchemy.sql import sqltypes
 from sqlalchemy.dialects import postgresql
-from sqlalchemy.dialects.postgresql.base import PGDialect
+from sqlalchemy.dialects.postgresql.base import PGDialect, ischema_names
+
+
+class XML(sqltypes.TypeEngine):
+    __visit_name__ = "XML"
+
+
+ischema_names["xml"] = XML


 @dt.dtype.register(PGDialect, postgresql.OID)
 def sa_pg_oid(_, sa_type, nullable=True):
     return dt.int32(nullable=nullable)
+
+
[email protected](PGDialect, XML)
+def sa_postgres_xml(_, satype, nullable=True):
+    return dt.Unknown(nullable=nullable)

Obviously this only prevents the CREATE VIEW exception. We still won't have XML support.

@helensilva14 helensilva14 added the priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) label Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) type: feature request 'Nice-to-have' improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

2 participants