Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(safe_array_union): allow for sorting nested structs #793

Merged
merged 3 commits into from
Sep 27, 2024

Conversation

project-defiant
Copy link
Contributor

@project-defiant project-defiant commented Sep 27, 2024

✨ Context

This PR closes opentargets/issues#3551

🛠 What does this PR implement

  • Addition of fields_order parameter to the safe_array_union function, so we can deal with cses when array contains unordered structs.
  • Implement fix for the vep parser when dealing with merge between two variantIndex objects merge of nested array fields.
  • Additional helper functions for resolving the column name from pyspark.sql.Column object and sorting structs within the array by fields_order.

🙈 Missing

🚦 Before submitting

  • Do these changes cover one single feature (one change at a time)?
  • Did you read the contributor guideline?
  • Did you make sure to update the documentation with your changes?
  • Did you make sure there is no commented out code in this PR?
  • Did you follow conventional commits standards in PR title and commit messages?
  • Did you make sure the branch is up-to-date with the dev branch?
  • Did you write any new necessary tests?
  • Did you make sure the changes pass local tests (make test)?
  • Did you make sure the changes pass pre-commit rules (e.g poetry run pre-commit run --all-files)?

Copy link
Contributor

@DSuveges DSuveges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crazy. Thanks for cleaning my mess. :D

@DSuveges DSuveges merged commit a135d26 into dev Sep 27, 2024
5 checks passed
@DSuveges DSuveges deleted the szsz-fix-vep-vi-schema-ordering branch September 27, 2024 14:37
return f.when(a.isNotNull() & b.isNotNull(), f.array_union(a, b)).otherwise(
f.coalesce(a, b)
)



Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this do the same as order_array_of_structs_by_field and order_array_of_structs_by_two_fields?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

safe_array_union handles incorrectly <array<struct>> when struct fields are not sorted
3 participants