DOC: add documentation about using shared libraries #700
Conversation
rgommers
force-pushed
the
doc-sharedlibs
branch
from
2dd3221
to
d254472
Compare
rgommers
force-pushed
the
doc-sharedlibs
branch
2 times, most recently
from
44f372e
to
89dbcc7
Compare
|
Okay, changed to @DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow. |
Why does the shared library have this funny name? |
| @@ -161,6 +161,13 @@ def test_local_lib(venv, wheel_link_against_local_lib): | |||
| assert int(output) == 3 | |||
|
|
|||
|
|
|||
| @pytest.mark.skipif(MESON_VERSION < (0, 64, 0), reason='Meson version too old') | |||
There was a problem hiding this comment.
What does break with older Meson versions?
There was a problem hiding this comment.
../meson.build:7:0: ERROR: python.find_installation got unknown keyword arguments "pure"
There was a problem hiding this comment.
Oh, this. We work around it in other test packages. I think we should either bump the minimum required Meson version or work around it here too passing the pure argument to the individual functions in the python module. Meson 0.64 was released November 6th 2022, thus almost 2 years ago. Maybe we can update the minimum required version to 0.64 without feeling bad 😃
There was a problem hiding this comment.
Yes, I'd prefer to bump to 0.64 rather than continue to use less idiomatic code. Especially since I plan to link to this test package from the docs as a complete and working example.
Seems like the fun of building on Windows - with MinGW we get |
rgommers
force-pushed
the
doc-sharedlibs
branch
2 times, most recently
from
ddaea0f
to
f95d160
Compare
| os.add_dll_directory(basedir) | ||
|
|
||
|
|
||
| _enable_sharedlib_loading() |
There was a problem hiding this comment.
Does this really need to define a function and immediately call it? Given that the setup code is reduced to three lines (that can be condensed to two) I think it is clearer to simply have the code directly at module level. Also, I think that it would be nice to have the code actually shown in the documentation page with a Sphinx code include directive. Then the function doc string can be included in the documentation page (I don't think it is easy to have it rendered nicely in the documentation page otherwise).
There was a problem hiding this comment.
Makes sense, if it stays at ~2 lines I'll drop the function. .. literalinclude:: is useful too, will do.
For completeness, delvewheel inserts this for external shared libraries that it vendors in:
# start delvewheel patch
def _delvewheel_patch_1_8_2():
import os
libs_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, 'numpy.libs'))
if os.path.isdir(libs_dir):
os.add_dll_directory(libs_dir)
_delvewheel_patch_1_8_2()
del _delvewheel_patch_1_8_2
# end delvewheel patchThe del is a nice touch if one wants to keep the namespace very clean.
|
Would it be worth to run |
| [project] | ||
| name = 'link-library-in-subproject' | ||
| version = '1.2.3' | ||
| license = 'MIT' |
There was a problem hiding this comment.
This makes the test dependent on pyproject-metadata 0.9.0 (the test fails on the CI job that tests with pyproject-metadata 0.8.0, which is currently the minimum required version)
There was a problem hiding this comment.
Yes, seeing the failure - I'll drop this line then.
There was a problem hiding this comment.
In #681 we are updating the minimum required version to 0.9.0. I would kip this.
I'll have a look, but if that's useful enough then I'd prefer to do it in a separate PR I think. |
Most of the packages I've worked with have only depended on system DLLs/shared libraries, but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy. The easy things to check are permissions (both shared libraries need execute permissions to work properly) and whether the From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up. I haven't run into if sys.platform == "cygwin":
def add_dll_directory(path: str):
os.environ["PATH"] = f"{os.environ['PATH']:s}:{path:s}"
os.add_dll_directory = add_dll_directorymight help. |
|
Thanks for the input @DWesl!
That changed recently, the SciPy 1.14.0 release has a shared library:
I'll try but I doubt it, since modifying
Is there a patch/repo somewhere for how the |
For NumPy: For SciPy, most recent I have:
It might still work, depending on what Windows thinks is the current directory for a |
Note that for Meson versions older than 1.2.0, CI failed with:
```
mesonpy.BuildError: Could not map installation path to an equivalent wheel directory: '{libdir_static}/libexamplelib.a'
```
because the `--skip-subprojects` install option isn't honored.
Hence the test skip on older versions.
rgommers
force-pushed
the
doc-sharedlibs
branch
from
f95d160
to
83e194b
Compare
|
The Cygwin tests pass now, thanks for the pointers @DWesl. Amending |
I think
It sounds like Windows does some helpful |
| ``delvewheel`` in order to make the built Python package usable locally. | ||
|
|
||
|
|
||
| Publishing wheels which depend on external shared libraries |
There was a problem hiding this comment.
There's no mention here of the topic of linking to libraries contained in other wheels. Even if it's not explained in the final draft, it should probably be mentioned and TODO added.
Opening this PR now to get feedback on topics and structure, not ready for detailed review yet.
We get a lot of questions about shared libraries, and this is a tricky thing to get right. So try to document how to use internal libraries as well as link to external shared libraries as well as possible. Subproject-related questions also come up more and more, and there are some extra gotchas here, so treat those as a third "source" of shared (or static) libraries.