fix: Prevent Oracle LOB columns from causing exceptions in Column and Row validation

[nj1973]

Oracle LOB columns are pretty restrictive to use, for example:

SQL> select count(col_clob) from pso_data_validator.dvt_ora2pg_types;
select count(col_clob) from pso_data_validator.dvt_ora2pg_types
             *
ERROR at line 1:
ORA-00932: inconsistent datatypes: expected - got CLOB

In DVT we lose the true data type and work with Ibis canonical types, for example a CLOB column becomes dt.String, a BLOB column becomes dt.Binary. This PR adds code to fetch the raw data types for Oracle tables/custom queries and excludes BLOB, CLOB and NCLOB from Column and Row (hash/concat) validations.

I only added the raw_metadata() for Oracle because that is the only engines we currently need it for. If we run into similar issues for other engines where we need to differentiate between specific raw data types then we can extend this technique further.

Ideally we would be able to add expressions to truly validate these types. I see that as a feature request rather than an exception avoidance issue which is what this one is.

[sundar-mudupalli-work]

Why skip this column when you can change the query builder here , using the ibis.cases method. Isn't that a simpler and better solution ?

Thanks.

Sundar Mudupalli

[nj1973]

I don't think I can skip it in the query builder because I don't have data type information for both the source and target. I believe ConfigManager is where I have all of the information required to exclude a column from both source and target queries.

[helensilva14]

Hi! Based on your discussion during the sync call, is this comment going to be resolved as-is or there will be code changes related to it?

[helensilva14]

All good from my side and only @sundar-mudupalli-work's comment is left to be resolved.

/gcbrun

[helensilva14]

@nj1973 - CLA check is failing because of my former Google email not being found anymore, sorry about that. I will need to go through the commits I did with the corp email and remove it from the authors

[helensilva14]

/gcbrun

[sundar-mudupalli-work]

Neil,

This PR is a bit confusing. I thought it was ensuring that exceptions were not thrown when CLOBs and BLOBs are row and column validated. However I find that column validation with clobs and blobs works well. Column validations correctly count the number of non null columns I believe.

This PR removes (count) column validations of LOB and BLOB columns. I believe min, max etc are also supported. These should likely stay as they are today.

The issue is with row validations, which throw an error. It would be nice to fix this and put out an info message. This PR fixes this and only puts out an INFO message if the verbose option is selected. We also have another case where an INFO message is output (when verbose option is selected) when a column is removed. In both cases, I believe an INFO message should be output - not just when verbose option is selected. I also opened another Issue to remove the verbose option entirely.

My general philosophy on validations (please feel free to disagree) is that the customer is trusting DVT to tell them when two tables are different. If we are excluding items from the validation - we should tell the customer explicitly that these are being excluded. We should not quietly exclude things and only disclose information when the verbose option is used.

Thank you for working on this.

Sundar Mudupalli

[nj1973]

I find that column validation with clobs and blobs works well. Column validations correctly count the number of non null columns I believe.

I retested the test case in the issue (#1335) and I still get the same exception:

sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00932: inconsistent datatypes: expected - got CLOB

BLOB columns will not have thrown an exception in the past because they are skilled due to this code (which I have replaced in this PR):

                if agg_type == "count":
                    # Oracle BLOB is invalid for use with SQL COUNT function.
                    return bool(
                        self.source_client.name == "oracle"
                        or self.target_client.name == "oracle"
                    )
                else:

[nj1973]

/gcbrun

[nj1973]

Closing this PR because it is based on a misunderstanding of current behaviour. I'll start over and maybe cherry pick any parts of these changes I need for second attempt.