feat: Polars build for Pyodide

[georgestagg]

Hi @ritchie46 , it was nice chatting to you recently.

As discussed, here is all the work so far for building a Polars wheel for Pyodide. As an update from when we spoke last: the required changes to LLVM have now hit nightly Rust and the inventory crate, so this should work from LLVM 19.1.6 and Rust nightly-2024-12-19 onwards. In the new CI for this PR, I have fixed the version of Rust nightly, but I believe after the next stable Rust release the pin can be dropped.

The PR makes a few changes, summarised below:

1. A couple of modifications have been made to deal with wasm32's native 32bit pointer width and using mimalloc under the emscripten OS (corresponding to Pyodide).

2. For the emscripten OS I setup the thread pool to use a single thread. Use of threads on Emscripten in combination with dynamic libraries is still a little shaky/experimental, and so this keeps things simple for now.

2b. For related reasons PyInProcessQuery is disabled for now when running under Pyodide.

3. I added a new "Test Pyodide" CI workflow. Currently it just "tests" by attempting to build the Pyodide wheel. Eventually we might want to load the wheel into Pyodide using node and run some real tests. I also added a new step in the "Release Python" workflow that builds the Pyodide wheel and adds it to a GitHub release. I would have just uploaded to PyPI with the other binaries but my understanding is that this does not work yet for Pyodide wheels.

3a. A set of features has been disabled when building for Pyodide in CI. Eventually we'd like to bring these back, but at the moment they pull in Rust dependencies that break the build. I expect this can all be improved except for the cloud related features, which might be difficult to get working in the Wasm sandbox.

3b. In a few places #[cfg(feature = "cloud")] has been added to conditionally compile some cloud-related code.

3c. In CI I disable the features using a janky sed, this might be doable entirely using Cargo.toml but I wasn't able to figure it out.

I have another copy of this branch, so if you'd like to make any further changes before merging please feel absolutely free to push extra commits directly to this PR branch.

If you'd like to experiment with the resulting Pyodide wheel, there's a copy available at https://github.com/georgestagg/polars/releases/tag/py-1.17.1 on my fork.

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.14%. Comparing base (ff00869) to head (5d7d1da).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-python/src/series/numpy_ufunc.rs	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20383      +/-   ##
==========================================
+ Coverage   78.38%   79.14%   +0.75%     
==========================================
  Files        1572     1566       -6     
  Lines      219962   219241     -721     
  Branches     2465     2465              
==========================================
+ Hits       172423   173512    +1089     
+ Misses      46971    45161    -1810     
  Partials      568      568              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[orlp]

Great work!

In the new CI for this PR, I have fixed the version of Rust nightly, but I believe after the next stable Rust release the pin can be dropped.

We already always pin the nightly version for Polars in our rust-toolchain.toml in the root directory. Is it not sufficient to simply bump the version inside that file to the newer version rather than doing it in the CI scripts?

[ritchie46]

Awesome @georgestagg. I shall see if we can bump the rustc version in general for the whole project. (We are always on nightly, so that's fine).

[ritchie46]

Thanks a lot @georgestagg. I cannot believe how smooth this went. (But you endured all the dark magic ofcourse ;) )