Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(python): Split file of lazyframe methods #19937

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

eitsupi
Copy link
Contributor

@eitsupi eitsupi commented Nov 23, 2024

As with DataFrame, it would be better to split the io-related functions into separate files for LazyFrame as well.

@github-actions github-actions bot added internal An internal refactor or improvement python Related to Python Polars labels Nov 23, 2024
@eitsupi eitsupi marked this pull request as draft November 23, 2024 05:11
@eitsupi eitsupi force-pushed the polars-python-lazyframe-io branch from 40e6d00 to 0bca62c Compare November 23, 2024 05:22
@eitsupi eitsupi marked this pull request as ready for review November 23, 2024 05:27
Copy link

codecov bot commented Nov 23, 2024

Codecov Report

Attention: Patch coverage is 99.34354% with 3 lines in your changes missing coverage. Please review.

Project coverage is 79.48%. Comparing base (414d883) to head (0bca62c).
Report is 120 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-python/src/lazyframe/io.rs 99.34% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main   #19937       +/-   ##
===========================================
+ Coverage   59.28%   79.48%   +20.19%     
===========================================
  Files        1555     1556        +1     
  Lines      216180   216181        +1     
  Branches     2456     2456               
===========================================
+ Hits       128155   171824    +43669     
+ Misses      87467    43799    -43668     
  Partials      558      558               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46
Copy link
Member

This on itself is fine, though we are not going to merge this because we want to revert all the splits. It is a requirement needed to get Polars working in WASM. See more context: PyO3/pyo3#2517

@ritchie46 ritchie46 closed this Dec 3, 2024
@eitsupi eitsupi deleted the polars-python-lazyframe-io branch December 3, 2024 21:16
@eitsupi eitsupi restored the polars-python-lazyframe-io branch December 3, 2024 22:43
@eitsupi
Copy link
Contributor Author

eitsupi commented Dec 3, 2024

This on itself is fine, though we are not going to merge this because we want to revert all the splits. It is a requirement needed to get Polars working in WASM. See more context: PyO3/pyo3#2517

@ritchie46 Thanks for the reply, but do that restriction really still exist today?
I know @georgestagg was working on builds for polars for pyodide, and from looking at its source code it seems to have been successfully built with multiple pymethods intact.
https://github.com/georgestagg/polars/releases

@georgestagg
Copy link
Contributor

georgestagg commented Dec 4, 2024

Indeed. My method works but requires using a custom LLVM & Rust compiler to support multiple pymethods with Wasm.

I'm working on upstreaming the changes required to LLVM here.

EDIT: The above PR is merged, but more work is still required. Another PR to LLVM will be needed, and then we must wait for the changes to hit rustc, amongst some other work. It may be that refactoring Polars to avoid multiple pymethods is faster, but I don't know for sure how much work that involves.

@eitsupi
Copy link
Contributor Author

eitsupi commented Dec 4, 2024

@georgestagg Thanks, amazing!

@ritchie46 Given that removing multiple pymethods would make the code significantly less readable, might it be better to keep the file split in favor of llvm/llvm-project#111008?

@eitsupi
Copy link
Contributor Author

eitsupi commented Dec 6, 2024

@ritchie46 Could you reopen this?

@ritchie46 ritchie46 reopened this Dec 7, 2024
@ritchie46
Copy link
Member

@georgestagg that's awesome. Could I contact you/ have a quick call? I'd like to learn more about this.

@eitsupi eitsupi force-pushed the polars-python-lazyframe-io branch from 0bca62c to 61b09e2 Compare December 8, 2024 13:00
@georgestagg
Copy link
Contributor

@ritchie46 Sure, feel free to contact me at georgestagg [at] gmail [dot] com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internal An internal refactor or improvement python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants