Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add str_length built-in check for PySpark backend #1709

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

marrov
Copy link

@marrov marrov commented Jun 25, 2024

Adds a builtin-check for str_length for the pyspark.sql backend, solving these issues:

Copy link

codecov bot commented Jun 25, 2024

Codecov Report

Attention: Patch coverage is 35.71429% with 9 lines in your changes missing coverage. Please review.

Project coverage is 82.47%. Comparing base (812b2a8) to head (e6db6fe).
Report is 111 commits behind head on main.

Files Patch % Lines
pandera/backends/pyspark/builtin_checks.py 35.71% 9 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (812b2a8) and HEAD (e6db6fe). Click for more details.

HEAD has 100 uploads more than BASE | Flag | BASE (812b2a8) | HEAD (e6db6fe) | |------|------|------| ||48|148|
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1709       +/-   ##
===========================================
- Coverage   94.28%   82.47%   -11.82%     
===========================================
  Files          91      117       +26     
  Lines        7013     8700     +1687     
===========================================
+ Hits         6612     7175      +563     
- Misses        401     1525     +1124     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cosmicBboy
Copy link
Collaborator

thanks @marrov ! please run pre-commit to fix linter errors: https://pandera.readthedocs.io/en/stable/CONTRIBUTING.html#set-up-pre-commit

@marrov
Copy link
Author

marrov commented Jun 27, 2024

thanks @marrov ! please run pre-commit to fix linter errors: https://pandera.readthedocs.io/en/stable/CONTRIBUTING.html#set-up-pre-commit

Yes, sorry for that I added it too hastily. Should be fixed now @cosmicBboy.

@cosmicBboy
Copy link
Collaborator

thanks @marrov ! sorry to extend this a little longer, but this check needs unit tests, which can live here: https://github.com/unionai-oss/pandera/blob/main/tests/pyspark/test_pyspark_check.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants