Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent column validation exceptions caused by Oracle CLOB/NCLOB columns used for JSON #1335

Closed
nj1973 opened this issue Nov 19, 2024 · 2 comments · Fixed by #1365
Closed
Assignees
Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@nj1973
Copy link
Contributor

nj1973 commented Nov 19, 2024

Test case

Oracle:

CREATE TABLE dvt_test.tab_json
(
  id VARCHAR2(10) NOT NULL PRIMARY KEY
, col_json1 CLOB
, col_json2 CLOB
);
ALTER TABLE dvt_test.tab_json ADD CONSTRAINT tab_json_chk1 CHECK (col_json1 IS JSON) ENABLE;
ALTER TABLE dvt_test.tab_json ADD CONSTRAINT tab_json_chk2 CHECK (col_json2 IS JSON) ENABLE;
INSERT INTO dvt_test.tab_json VALUES (1,'{"a": 1}','{"a": 1}');
COMMIT;

PostgreSQL:

CREATE TABLE dvt_test.tab_json
(
  id varchar(10) NOT NULL PRIMARY KEY
, col_json1 json
, col_json2 jsonb
);
INSERT INTO dvt_test.tab_json VALUES (1,'{"a": 1}','{"a": 1}');

Column validation
DVT command Oracle vs PostgreSQL:

$ data-validation validate column -sc ora_local -tc pg_local -tbls="dvt_test.tab_json" --count '*' --sum '*' --min '*' --max '*' --avg '*' --std '*' --filter-status fail
...
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function length(json) does not exist
LINE 2: ..._json2 AS col_json2, length(t1.id) AS length__id, length(t1....
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The problem is that Oracle stores JSON data in a CLOB.

Row validation
DVT command Oracle vs PostgreSQL:

$ data-validation validate row -sc ora_local -tc pg_local -tbls="dvt_test.tab_json" -pk=id --hash=col_json1,col_json2
...
  File "/usr/local/google/home/neiljohnson/github/professional-services-data-validator/.venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
cx_Oracle.DatabaseError: ORA-00932: inconsistent datatypes: expected CLOB got CHAR
@nj1973 nj1973 added type: feature request 'Nice-to-have' improvement, new feature or different behavior or design. priority: p1 High priority. Fix may be included in the next release. labels Nov 19, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Nov 19, 2024

We should add the test columns above to the dvt_ora2pg_types integration test.

@nj1973 nj1973 self-assigned this Nov 19, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Nov 21, 2024

I feel this issue needs to be split into two.

  1. Prevent Oracle CLOB/NCLOB from generating exceptions when used in a normal context. This is the higher priority part to prevent command failures.
  2. Look at general JSON support and tests across all validation types. Including spotting Oracle CLOB/NCLOB used for JSON data. This is lower priority feature request.

I will complete item 1 on this issue now, I've opened #1338 for item 2 above.

@nj1973 nj1973 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed type: feature request 'Nice-to-have' improvement, new feature or different behavior or design. labels Nov 21, 2024
@nj1973 nj1973 changed the title PostgreSQL jsonb data type support Prevent exceptions caused by Oracle CLOB/NCLOB/BLOB columns Nov 28, 2024
@nj1973 nj1973 changed the title Prevent exceptions caused by Oracle CLOB/NCLOB/BLOB columns Prevent exceptions caused by Oracle CLOB/NCLOB columns used for JSON Dec 4, 2024
@nj1973 nj1973 changed the title Prevent exceptions caused by Oracle CLOB/NCLOB columns used for JSON Prevent column validation exceptions caused by Oracle CLOB/NCLOB columns used for JSON Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
1 participant