DRAFT: Add provision sqlalchemy #243
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 20 commits
October 6, 2023 11:57
Necessary because it breaks some namespace imports in SQLAlchemy "types.py" is a pretty common package name. Other dialects appear to do the same underscore prefix approach Signed-off-by: Jesse Whitehouse <[email protected]>
…t them Neither inline nor bound parameter implementations can handle BINARY test_suite.py::BinaryTest_databricks+databricks::test_binary_roundtrip[7\xe7\x9f] SKIPPED (pysql doesn't support binding of BINARY type parameters) test_suite.py::BinaryTest_databricks+databricks::test_binary_roundtrip[this is binary] SKIPPED (pysql doesn't support binding of BINARY type parameters) Signed-off-by: Jesse Whitehouse <[email protected]>
This test requires date_historic and datetime_literals requirements be marked open. test_suite.py::DateHistoricTest_databricks+databricks::test_literal PASSED test_suite.py::DateHistoricTest_databricks+databricks::test_null PASSED test_suite.py::DateHistoricTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateHistoricTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateHistoricTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::DateHistoricTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
No changes to requirements.py were needed test_suite.py::DateTest_databricks+databricks::test_literal PASSED test_suite.py::DateTest_databricks+databricks::test_null PASSED test_suite.py::DateTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::DateTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
This is the first case where implement our own process_result_value method. For this to work, we need to update the dialect.colspecs map so that SQLAlchemy knows to use our custom implementation rather than its default test_suite.py::DateTimeHistoricTest_databricks+databricks::test_literal PASSED test_suite.py::DateTimeHistoricTest_databricks+databricks::test_null PASSED test_suite.py::DateTimeHistoricTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateTimeHistoricTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateTimeHistoricTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::DateTimeHistoricTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
These were fixed by updating our @compiles directives and dialect.colspecs test_suite.py::DateTimeTest_databricks+databricks::test_literal PASSED test_suite.py::DateTimeTest_databricks+databricks::test_null PASSED test_suite.py::DateTimeTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateTimeTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateTimeTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::DateTimeTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
This fix follows the same pattern used to make DateTimeHistoricTest work test_suite.py::TimeTest_databricks+databricks::test_literal PASSED test_suite.py::TimeTest_databricks+databricks::test_null PASSED test_suite.py::TimeTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::TimeTest_databricks+databricks::test_round_trip PASSED test_suite.py::TimeTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::TimeTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
The fixes for DateTimes in the past few commits have enabled these tests to pass. test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_literal PASSED test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_null PASSED test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::DateTimeCoercedToDateTimeTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
I needed to add an entry to requirements.py otherwise these tests were skipped. test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_literal PASSED test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_null PASSED test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_round_trip PASSED test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::TimestampMicrosecondsTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
Changes in earlier commits around DateTime types have enabled these to pass. test_suite.py::DateTimeMicrosecondsTest_databricks+databricks::test_literal PASSED test_suite.py::DateTimeMicrosecondsTest_databricks+databricks::test_null PASSED test_suite.py::DateTimeMicrosecondsTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::DateTimeMicrosecondsTest_databricks+databricks::test_round_trip PASSED test_suite.py::DateTimeMicrosecondsTest_databricks+databricks::test_round_trip_decorated PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
I've found that test_dont_truncate_rightside test is flaky because sometimes DBR returns the correct data but in the wrong order. It's supposed to check that a query returns ["AB", "BC"] but sometimes it returns ["BC", "AB"] which is the right data in the wrong order. Python list comparison doesn't evaluate that ["AB", "BC"] == ["BC", "AB"]. We could reimplement the test using collections.Counter in stdlib or check with DBR team about why this order is sometimes different. Signed-off-by: Jesse Whitehouse <[email protected]> --- Also, we had to break with SQLAlchemy's advice to never implement the TypeDecorator.literal_processor method because otherwise our strings end up double-escaped and raise a syntax error. test_suite.py::StringTest_databricks+databricks::test_concatenate_binary PASSED test_suite.py::StringTest_databricks+databricks::test_concatenate_clauselist PASSED test_suite.py::StringTest_databricks+databricks::test_dont_truncate_rightside[%B%-expected0] PASSED test_suite.py::StringTest_databricks+databricks::test_dont_truncate_rightside[A%C%Z-expected2] PASSED test_suite.py::StringTest_databricks+databricks::test_dont_truncate_rightside[A%C-expected1] PASSED test_suite.py::StringTest_databricks+databricks::test_literal PASSED test_suite.py::StringTest_databricks+databricks::test_literal_backslashes PASSED test_suite.py::StringTest_databricks+databricks::test_literal_non_ascii PASSED test_suite.py::StringTest_databricks+databricks::test_literal_quoting PASSED test_suite.py::StringTest_databricks+databricks::test_nolength_string PASSED
The fixes from fixing StringTest also fixed these tests test_suite.py::TextTest_databricks+databricks::test_literal PASSED test_suite.py::TextTest_databricks+databricks::test_literal_backslashes PASSED test_suite.py::TextTest_databricks+databricks::test_literal_non_ascii PASSED test_suite.py::TextTest_databricks+databricks::test_literal_percentsigns PASSED test_suite.py::TextTest_databricks+databricks::test_literal_quoting PASSED test_suite.py::TextTest_databricks+databricks::test_text_empty_strings PASSED test_suite.py::TextTest_databricks+databricks::test_text_null_strings PASSED test_suite.py::TextTest_databricks+databricks::test_text_roundtrip PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
I had to update our custom Time() override to cope with both non-microseconds timestrings and microseconds time strings. So I reran the TimeTest() as well to confirm both types still work. test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_literal PASSED test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_null PASSED test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_null_bound_comparison PASSED test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_round_trip PASSED test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_round_trip_decorated PASSED test_suite.py::TimeMicrosecondsTest_databricks+databricks::test_select_direct PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
I added exclusions to requirements.py for features which Databricks doesn't
natively support. These tests apply to the generic (camelcase) Numeric() type.
Therefore the tests are applicable to the DECIMAL Databricks data type. Some of
these features _will_ work for Databricks' FLOAT type. Those tests marked as skipped
in the below listing are skipped because of the exclusions called out in requirements.py
test_suite.py::NumericTest_databricks+databricks::test_decimal_coerce_round_trip PASSED
test_suite.py::NumericTest_databricks+databricks::test_decimal_coerce_round_trip_w_cast PASSED
test_suite.py::NumericTest_databricks+databricks::test_enotation_decimal SKIPPED ('test/test_suite.py::NumericTest_databricks+databricks::test_enotation_decimal (call)' : marked as skip)
test_suite.py::NumericTest_databricks+databricks::test_enotation_decimal_large SKIPPED ('test/test_suite.py::NumericTest_databricks+databricks::test_enotation_decimal_large (call)' : marked as skip)
test_suite.py::NumericTest_databricks+databricks::test_float_as_decimal PASSED
test_suite.py::NumericTest_databricks+databricks::test_float_as_float PASSED
test_suite.py::NumericTest_databricks+databricks::test_float_coerce_round_trip SKIPPED ('test/test_suite.py::NumericTest_databricks+databricks::test_float_coerce_round_trip (call)' : marked as skip)
test_suite.py::NumericTest_databricks+databricks::test_float_custom_scale SKIPPED ('test/test_suite.py::NumericTest_databricks+databricks::test_float_custom_scale (call)' : marked as skip)
test_suite.py::NumericTest_databricks+databricks::test_float_is_not_numeric[Double] PASSED
test_suite.py::NumericTest_databricks+databricks::test_float_is_not_numeric[Float] PASSED
test_suite.py::NumericTest_databricks+databricks::test_infinity_floats PASSED
test_suite.py::NumericTest_databricks+databricks::test_many_significant_digits SKIPPED ('test/test_suite.py::NumericTest_databricks+databricks::test_many_significant_digits (call)' : marked as skip)
test_suite.py::NumericTest_databricks+databricks::test_numeric_as_decimal PASSED
test_suite.py::NumericTest_databricks+databricks::test_numeric_as_float PASSED
test_suite.py::NumericTest_databricks+databricks::test_numeric_no_decimal PASSED
test_suite.py::NumericTest_databricks+databricks::test_numeric_null_as_decimal PASSED
test_suite.py::NumericTest_databricks+databricks::test_numeric_null_as_float PASSED
test_suite.py::NumericTest_databricks+databricks::test_precision_decimal PASSED
test_suite.py::NumericTest_databricks+databricks::test_render_literal_float PASSED
test_suite.py::NumericTest_databricks+databricks::test_render_literal_numeric PASSED
test_suite.py::NumericTest_databricks+databricks::test_render_literal_numeric_asfloat PASSED
Signed-off-by: Jesse Whitehouse <[email protected]>
These tests appear to be fixed by fixing our @compiles decorators test_suite.py::BooleanTest_databricks+databricks::test_null PASSED test_suite.py::BooleanTest_databricks+databricks::test_render_literal_bool PASSED test_suite.py::BooleanTest_databricks+databricks::test_round_trip PASSED test_suite.py::BooleanTest_databricks+databricks::test_whereclause PASSED Signed-off-by: Jesse Whitehouse <[email protected]>
…suite fails. This appears to be a programming error in SQLAlchemy Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
… file Signed-off-by: Jesse Whitehouse <[email protected]>
…ance tests. Setting this up isn't documented anywhere and the implementation here doesn't actually work because the test runner keeps trying to create tables within the catalog.schema in the connection string. Need to research more about why this happens. But wanted to stash this code for now as we'll want to address this for faster compliance test runs in the future. Signed-off-by: Jesse Whitehouse <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DRAFT: This scaffolds in the code for running parallel dialect compliance
tests. Setting this up isn't documented anywhere and the implementation here
doesn't actually work because the test runner keeps trying to create
tables within the catalog.schema in the connection string.
Need to research more about why this happens. But wanted to stash this code
for now as we'll want to address this for faster compliance test runs
in the future.