add stubs for dlopen, dlsym, etc. #443

dicej · 2023-10-20T16:31:33Z

This adds weak exports for the POSIX dlopen, dlsym, dlclose, and dlerror functions, allowing code which uses those features to compile. The implementations are stubs which always fail since there is currently no official standard for runtime dynamic linking.

Since the symbols are weak, they can be overriden with useful, runtime-specific implementations, e.g. based on host functions or statically-generated tables (see https://github.com/dicej/component-linking-demo for an example of the latter).

This adds weak exports for the POSIX `dlopen`, `dlsym`, `dlclose`, and `dlerror` functions, allowing code which uses those features to compile. The implementations are stubs which always fail since there is currently no official standard for runtime dynamic linking. Since the symbols are weak, they can be overriden with useful, runtime-specific implementations, e.g. based on host functions or statically-generated tables (see https://github.com/dicej/component-linking-demo for an example of the latter). Signed-off-by: Joel Dice <[email protected]>

abrown

I'm also fine merging this as-is, but wanted to get some feedback on the following: it feels like this pattern, "provide some maybe-working, weak symbols," is what the libwasi-emulated-*.a libraries are for. @sunfishcode, @sbc100: any opinions on this?

sbc100 · 2023-10-26T02:40:21Z

In order to override the weak symbols one would need to provide strong symbols, which already works today doesn't it (i.e. without this change). What is the advantage of having these failing weak symbols?

dicej · 2023-10-27T21:34:11Z

In order to override the weak symbols one would need to provide strong symbols, which already works today doesn't it (i.e. without this change). What is the advantage of having these failing weak symbols?

(Sorry for being slow to reply; busy week)

The main advantage is being able to compile projects that use dlopen, dlsym, etc. using wasi-sdk without adding CFLAGS or LDFLAGS to the build to point to additional header files and libraries beyond what wasi-libc provides.

sbc100 · 2023-10-27T22:43:00Z

In order to override the weak symbols one would need to provide strong symbols, which already works today doesn't it (i.e. without this change). What is the advantage of having these failing weak symbols?

(Sorry for being slow to reply; busy week)

The main advantage is being able to compile projects that use dlopen, dlsym, etc. using wasi-sdk without adding CFLAGS or LDFLAGS to the build to point to additional header files and libraries beyond what wasi-libc provides.

I would imagine that most projects that use dlopen/dlsym have a code path that disables them? e.g. HAVE_DLOPEN. Wouldn't it be better to use that than to link against stub implementations?

Do you have a examples of a codebases that would benefit from these stubs? i.e. Do not have a way to disable the dlopen code path yet gracefully handle the failure of dlopen?

dicej · 2023-10-27T23:09:28Z

The main advantage is being able to compile projects that use dlopen, dlsym, etc. using wasi-sdk without adding CFLAGS or LDFLAGS to the build to point to additional header files and libraries beyond what wasi-libc provides.

I would imagine that most projects that use dlopen/dlsym have a code path that disables them? e.g. HAVE_DLOPEN. Wouldn't it be better to use that than to link against stub implementations?

Do you have a examples of a codebases that would benefit from these stubs? i.e. Do not have a way to disable the dlopen code path yet gracefully handle the failure of dlopen?

Ah, but I don't want to disable them. I've actually forked CPython to enable dlopen for WASI, and I'd like to upstream that change. I very much want CPython to use dlopen, and in fact it's the whole reason for the wasi-sdk and wasi-libc PRs I've opened to date. Also, it doesn't need to handle the failure of dlopen because it won't fail -- I'll provide the correct, working implementation when linking with wasm-tools component link and the libc version will never actually be used.

Beyond CPython, I want to be able to compile any library that uses dlopen for WASI and use wasm-tools component link to provide the appropriate implementation so it works at runtime. Alternatively, yamt mentioned an implementation that uses host functions to handle linking: https://github.com/yamt/toywasm/tree/master/examples/libdl.

Granted, if you don't have a working dlopen implementation in mind (such as one of the above), you probably do want to disable its use at build time, and of course you'll still be able to do that if the library in question supports it.

sbc100 · 2023-10-27T23:13:57Z

The main advantage is being able to compile projects that use dlopen, dlsym, etc. using wasi-sdk without adding CFLAGS or LDFLAGS to the build to point to additional header files and libraries beyond what wasi-libc provides.

I would imagine that most projects that use dlopen/dlsym have a code path that disables them? e.g. HAVE_DLOPEN. Wouldn't it be better to use that than to link against stub implementations?
Do you have a examples of a codebases that would benefit from these stubs? i.e. Do not have a way to disable the dlopen code path yet gracefully handle the failure of dlopen?

Ah, but I don't want to disable them. I've actually forked CPython to enable dlopen for WASI, and I'd like to upstream that change. I very much want CPython to use dlopen, and in fact it's the whole reason for the wasi-sdk and wasi-libc PRs I've opened to date. Also, it doesn't need to handle the failure of dlopen because it won't fail -- I'll provide the correct, working implementation when linking with wasm-tools component link and the libc version will never actually be used.

Sorry I must be misunderstanding something. If you are going to provide a working implementation at link time I don't see why this change is useful. Can't you already do that without this change?

sbc100 · 2023-10-27T23:19:31Z

The main advantage is being able to compile projects that use dlopen, dlsym, etc. using wasi-sdk without adding CFLAGS or LDFLAGS to the build to point to additional header files and libraries beyond what wasi-libc provides.

I would imagine that most projects that use dlopen/dlsym have a code path that disables them? e.g. HAVE_DLOPEN. Wouldn't it be better to use that than to link against stub implementations?
Do you have a examples of a codebases that would benefit from these stubs? i.e. Do not have a way to disable the dlopen code path yet gracefully handle the failure of dlopen?

Ah, but I don't want to disable them. I've actually forked CPython to enable dlopen for WASI, and I'd like to upstream that change. I very much want CPython to use dlopen, and in fact it's the whole reason for the wasi-sdk and wasi-libc PRs I've opened to date. Also, it doesn't need to handle the failure of dlopen because it won't fail -- I'll provide the correct, working implementation when linking with wasm-tools component link and the libc version will never actually be used.

Sorry I must be misunderstanding something. If you are going to provide a working implementation at link time I don't see why this change is useful. Can't you already do that without this change?

I think I see now.. you are talking about that case where libc itself is a dynamic library.. and you want to link against one version of libc (that doesn't have dlopen) but then run with version that does have it? I.e. you are going to run against a different version of libc.so to the one that you linked against.

Presumably that also means that this patch is really only useful in the case the libc is dynamically linked?

I'm still not 100% convinced this is that right way forward.. but I think I'm seeing you use case now. What do others thing about this approach? @sunfishcode ?

dicej · 2023-10-27T23:21:30Z

Yes, that's correct. And I'm happy to restrict this to the dynamically linked version of libc if that's feasible.

dicej · 2023-10-27T23:23:28Z

Oops, I should have read more carefully. I'm actually going to use the exact same libc.so in both cases, but also link in my special libdl-for-shared-everything-linking.so so that its version of dlopen etc. is used instead of libc.so's version.

dicej · 2023-10-27T23:27:29Z

Here's an excerpt from the Makefile I'm using that does this, in case it's helpful:

$(BUILD_DIR)/bar.wasm: \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(NUMPY_LIBRARIES) \
		$(WASI_ADAPTER)
	$(WASM_TOOLS_CLI) component link \
		--adapt wasi_snapshot_preview1=$(WASI_ADAPTER) \
		--dl-openable $(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(shell echo $(NUMPY_LIBRARIES) | \
			sed 's_$(BUILD_DIR)/\([^ ]*\)_--dl-openable /build/\1=$(BUILD_DIR)/\1_g') \
		-o $@

In this case, libdl.so is customized for how wasm-tools component link supports "dynamic linking" to other modules in the component I'm producing. Note that libdl.so comes before $(LIBC) in the linking order, so it's version of dlopen etc. takes precedence.

yamt · 2023-11-07T06:44:36Z

Here's an excerpt from the Makefile I'm using that does this, in case it's helpful:

$(BUILD_DIR)/bar.wasm: \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(NUMPY_LIBRARIES) \
		$(WASI_ADAPTER)
	$(WASM_TOOLS_CLI) component link \
		--adapt wasi_snapshot_preview1=$(WASI_ADAPTER) \
		--dl-openable $(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(shell echo $(NUMPY_LIBRARIES) | \
			sed 's_$(BUILD_DIR)/\([^ ]*\)_--dl-openable /build/\1=$(BUILD_DIR)/\1_g') \
		-o $@

In this case, libdl.so is customized for how wasm-tools component link supports "dynamic linking" to other modules in the component I'm producing. Note that libdl.so comes before $(LIBC) in the linking order, so it's version of dlopen etc. takes precedence.

i'm a bit confused. do you mean you don't really need these stubs?

yamt · 2023-11-07T06:37:02Z

expected/wasm32-wasi-threads/predefined-macros.txt

+#define RTLD_DI_LINKMAP 2
+#define RTLD_GLOBAL 256
+#define RTLD_LAZY 1
+#define RTLD_LOCAL 0


isn't it straightforward to use a non-zero value?
an implementation can still choose to ignore it.

@dicej ping

Sorry, I didn't notice these comments earlier. The values in these files come straight from libc-top-half/musl/include/dlfcn.h, which appears to remain unchanged since it was imported from musl libc. glibc seems to use the same values for these constants also. I'm not necessarily opposed to changing the values, but sticking with the musl values seems like a safe default here.

Today I learned that RTLD_GLOBAL is the default on MacOS, but RTLD_LOCAL is the default on Linux (which I guess is why RTLD_LOCAL is zero on musl and glibc but nonzero on MacOS). I guess we ought to decide that the default should be for wasi-libc if we ever ship more than a stub implementation.

unless you want to decide the default right now, we can postpone the decision by making it a non-zero value.

I just pushed an update changing it to a non-zero value.

I'm not sure I understand how we can make both of these non-zero. This is a binary choice between global or local symbol binding, its not possible to choose neither. The question is what the default should be when neither is specified and the answer has to be one or the other.

I suggest we go with that musl's defaults for now given that we are largely a port of musl here.

Oh wait, I understand now. Please ignore my last comment. Having RTLD_LOCAL and RTLD_GLOBAL both be non-zero allows the implementation to detect the default case and choose one or the other at runtime. SGTM

yamt · 2023-11-07T06:39:27Z

expected/wasm32-wasi-threads/predefined-macros.txt

@@ -1499,6 +1499,15 @@
 #define RRFIXEDSZ NS_RRFIXEDSZ
 #define RRQ 01
 #define RS_HIPRI 0x01
+#define RTLD_DEFAULT ((void *)0)
+#define RTLD_DI_LINKMAP 2


does this make sense w/o dlinfo?

@dicej ping

Yeah, I don't know why this is defined unconditionally in dlfcn.h. We could move it into the #if defined(_GNU_SOURCE) || defined(_BSD_SOURCE) block in that file if desired.

I just pushed an update to address this.

sbc100 · 2023-11-07T16:52:02Z

Here's an excerpt from the Makefile I'm using that does this, in case it's helpful:
$(BUILD_DIR)/bar.wasm: \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(NUMPY_LIBRARIES) \
		$(WASI_ADAPTER)
	$(WASM_TOOLS_CLI) component link \
		--adapt wasi_snapshot_preview1=$(WASI_ADAPTER) \
		--dl-openable $(BUILD_DIR)/libfoo.so \
		$(BUILD_DIR)/libbar.so \
		$(BUILD_DIR)/libdl.so \
		$(BUILD_DIR)/libpython3.11.so \
		$(LIBC) \
		$(LIBCXX) \
		$(LIBCXXABI) \
		$(shell echo $(NUMPY_LIBRARIES) | \
			sed 's_$(BUILD_DIR)/$[^ ]*$_--dl-openable /build/\1=$(BUILD_DIR)/\1_g') \
		-o $@
In this case, libdl.so is customized for how wasm-tools component link supports "dynamic linking" to other modules in the component I'm producing. Note that libdl.so comes before $(LIBC) in the linking order, so it's version of dlopen etc. takes precedence.
i'm a bit confused. do you mean you don't really need these stubs?

IIRC that use case if dynamic linking. They want link (at build time) against a .so contain the stubs but then at runtime link against a library that contains real implementations.

dicej · 2023-11-08T01:21:40Z

It might be helpful to step back and look at the big picture here. Many (most?) popular high-level languages have some way to do FFI calls to C ABI native code loaded via dlopen/dlsym or the equivalent. Java has JNI, Ruby has extension libraries, .NET has DllImport, Python has native extensions, etc.

Python is perhaps the most notable of these since many popular packages rely heavily on native extensions. Those packages are distributed as platform-specific "wheels", each of which includes the Python code plus one or more shared libraries built for that specific platform. The wheels are usually published to pypi.org, the default package index and repository used by pip, a popular package manager for Python.

The immediate motivation behind #429 and WebAssembly/wasi-sdk#338 was to lay the foundation for introducing a new WASI platform for Python wheels alongside the existing ones such as MacOS/ARM64 and Windows/x64. Such wheels would contain shared libraries built using a future release of wasi-sdk and be compatible with any build of CPython built using the same version of wasi-sdk (which is the best we can currently offer given that wasi-libc does not yet make any backwards compatibility guarantees).

In order to actually use the shared libraries in these wheels, CPython needs some way to load them, and the easiest way is to build it with dlopen/dlsym enabled. The idea behind this PR is to make that easy to do. The dlfcn.h header and dl.c stubs would allow a dlopen-enabled WASI CPython to be built with a stock, unmodified wasi-sdk and wasi-libc. The alternative would be for CPython to provide its own, built-in stubs for these functions. Likewise, other language runtimes, applications, or libraries which want to use dlopen/dlsym when targeting WASI would be responsible for providing stubs to compile against. In the case of a library that doesn't have any special WASI support, the user of that library would be responsible for providing the stubs and injecting them via the appropriate CFLAGS and LDFLAGS when building the library. I'd like to avoid that putting that burden on third party developers and users, hence this PR.

As I mentioned earlier, the intention here is that the stubs will be replaced (either at runtime or component composition time) with a working implementation appropriate to the environment in which they're used. One such implementation might load libraries Emscripten-style, via a URL. Another might search the filesystem. wasm-tools component link can do shared-everything linking to bundle everything into a self-contained component, with optional support for symbol lookup via dlopen. Eventually wasm-tools component link will also support creating components that import modules (e.g. from a registry) as an alternative to bundling.

To summarize: my goal is to make it easy to create and use WASI shared libraries -- including via dlopen -- in all contexts where that's useful, and Python happens to be one of the more urgent cases. If there are better ways to achieve that which provides a comparable level of convenience to wasi-sdk users, I'm happy to consider them.

yamt · 2023-11-08T02:26:41Z

my impression is that it's a bit tricky to replace only a part of a shared library. (libc.so)
on the other hand, i can understand it's useful to provide a stub implementation here.

how about providing a stub libdl.so as a separate library from libc?
that way, an executable using dlopen can be run more naturally as it has an explicit record in needed_dynlibs. (otherwise you would need an LD_PRELOAD equivalent functionality to load your version of dlopen implementation explicitly.)
as requiring -ldl for dlopen is not too uncommon, i guess it isn't a problem for applications like python.

dicej · 2023-11-08T02:38:51Z

how about providing a stub libdl.so as a separate library from libc?

Sounds good to me. Ideally I'd like to include it as part of wasi-sdk (i.e. as part of wasi-libc or some other dependency of wasi-sdk). Any objections to me adding a libdl.so to the wasi-libc makefile alongside libc.so and libwasi-emulated-*.so? @sbc100 does that sound okay to you?

sbc100 · 2023-11-08T05:18:16Z

how about providing a stub libdl.so as a separate library from libc?

Sounds good to me. Ideally I'd like to include it as part of wasi-sdk (i.e. as part of wasi-libc or some other dependency of wasi-sdk). Any objections to me adding a libdl.so to the wasi-libc makefile alongside libc.so and libwasi-emulated-*.so? @sbc100 does that sound okay to you?

That does sounds little more logical I guess, since then you replacing the entire libdl.so and not just part of libc.so.

I guess we could even make this a dynamic-only library since there would not be any use the static one. But that also begs the question, why not just supply your working libdl.so and link time and link against that, which would take wasi-libc out of the picture completely. I guess one argument would be that there could be other users who could benefit from a shared libdl.so.. not just python.. but I'm not sure we are that stage yet.

I don't feel strongly either way.

dicej · 2023-11-08T14:52:49Z

I guess we could even make this a dynamic-only library since there would not be any use the static one. But that also begs the question, why not just supply your working libdl.so and link time and link against that, which would take wasi-libc out of the picture completely. I guess one argument would be that there could be other users who could benefit from a shared libdl.so.. not just python.. but I'm not sure we are that stage yet.

Yes, there are three options here with respect to CPython:

I can maintain a fork of CPython indefinitely, patched to enable dlopen, and either build it with CFLAGS=-I/path-to-dlfcn-h/ and LDFLAGS="-L/path-to-libdl-so/ -ldl or bake those flags into the build system. Anyone who wants to use this version would need to either build my fork from source or download a pre-built artifact from my GitHub releases.
If they're willing, the CPython folks can enable dlopen for WASI in the official repo using a built-in stub libdl.so. Then people would be able to build from the official source or use the official builds (once WASI becomes a tier 2 supported platform, which is what Brett Cannon is working towards).
We merge this PR, and the CPython folks adopt the next wasi-sdk release and enable dlopen for WASI. In this case the stub is part of wasi-libc so CPython need not provide it. Furthermore, any runtime, app, or library that uses dlopen will benefit -- not just CPython. I'm personally planning to work on bringing Java/JNI and Ruby with extension libraries to WASI, and I'd prefer to avoid copying a stub library around and explaining to the upstream projects why it is necessary.

sbc100 · 2023-11-08T15:44:08Z

2. If they're willing, the CPython folks can enable dlopen for WASI in the official repo using a built-in stub libdl.so. Then people would be able to build from the official source or use the official builds (once WASI becomes a tier 2 supported platform, which is what Brett Cannon is working towards).

Does CPython gracefully handle failure of dlopen at runtime like this? i.e. You are telling cpython that dlopen exists but in at runtime it might still be a stub (unless that user happens to interpose a working version like you will). I wonder if its better to declare/detect dlopen missing at build time for some projects (maybe CPython is one of them?).

Or are you imagining that all users of CPython+wasi is inject a working version of dlopen at runtime? (Presumably those who build without dyanmic linking have no way to do this).

dicej · 2023-11-08T16:40:56Z

Does CPython gracefully handle failure of dlopen at runtime like this? i.e. You are telling cpython that dlopen exists but in at runtime it might still be a stub (unless that user happens to interpose a working version like you will). I wonder if its better to declare/detect dlopen missing at build time for some projects (maybe CPython is one of them?).

Yes, it does handle it gracefully -- the same way it would if it couldn't find the native extension library on the filesystem. I.e. it will behave the same way it does today: trying to load a native extension will fail because it is running in an environment where loading native extensions is not possible.

BTW, keep in mind that 99% of users will not be building CPython from source -- they will be using an official, pre-built release, in which case build-time feedback isn't relevant to them.

Or are you imagining that all users of CPython+wasi is inject a working version of dlopen at runtime? (Presumably those who build without dyanmic linking have no way to do this).

All CPython users who want to use native extensions with CPython+wasi will inject a working version of dlopen, yes (though not necessarily at runtime -- possibly at composition time). componentize-py does this automatically today, and it can be done manually using wasm-tools component link. Other implementations might do it at runtime with an empscripten-style trampoline or with a host function like toywasm does.

sbc100 · 2023-11-08T16:50:17Z

Does CPython gracefully handle failure of dlopen at runtime like this? i.e. You are telling cpython that dlopen exists but in at runtime it might still be a stub (unless that user happens to interpose a working version like you will). I wonder if its better to declare/detect dlopen missing at build time for some projects (maybe CPython is one of them?).

Yes, it does handle it gracefully -- the same way it would if it couldn't find the native extension library on the filesystem. I.e. it will behave the same way it does today: trying to load a native extension will fail because it is running in an environment where loading native extensions is not possible.

BTW, keep in mind that 99% of users will not be building CPython from source -- they will be using an official, pre-built release, in which case build-time feedback isn't relevant to them.

Or are you imagining that all users of CPython+wasi is inject a working version of dlopen at runtime? (Presumably those who build without dyanmic linking have no way to do this).

All CPython users who want to use native extensions with CPython+wasi will inject a working version of dlopen, yes (though not necessarily at runtime -- possibly at composition time). componentize-py does this automatically today, and it can be done manually using wasm-tools component link. Other implementations might do it at runtime with an empscripten-style trampoline or with a host function like toywasm does.

SGTM.. sounds like a good plan.

Out of interest when you talk about injecting a working version of "dlopen" at "composition time" is that just the "libdl" functions that get linked in, or are you also talking about the shared libraries that are opened by "dlopen" ? Are they somehow determined ahead of time and kind of pre-linked? i.e. is your dlopen actually able to load stuff at runtime off the wire?

sbc100 · 2023-11-08T16:51:05Z

Does CPython gracefully handle failure of dlopen at runtime like this? i.e. You are telling cpython that dlopen exists but in at runtime it might still be a stub (unless that user happens to interpose a working version like you will). I wonder if its better to declare/detect dlopen missing at build time for some projects (maybe CPython is one of them?).

Yes, it does handle it gracefully -- the same way it would if it couldn't find the native extension library on the filesystem. I.e. it will behave the same way it does today: trying to load a native extension will fail because it is running in an environment where loading native extensions is not possible.
BTW, keep in mind that 99% of users will not be building CPython from source -- they will be using an official, pre-built release, in which case build-time feedback isn't relevant to them.

Or are you imagining that all users of CPython+wasi is inject a working version of dlopen at runtime? (Presumably those who build without dyanmic linking have no way to do this).

All CPython users who want to use native extensions with CPython+wasi will inject a working version of dlopen, yes (though not necessarily at runtime -- possibly at composition time). componentize-py does this automatically today, and it can be done manually using wasm-tools component link. Other implementations might do it at runtime with an empscripten-style trampoline or with a host function like toywasm does.

SGTM.. sounds like a good plan.

Out of interest when you talk about injecting a working version of "dlopen" at "composition time" is that just the "libdl" functions that get linked in, or are you also talking about the shared libraries that are opened by "dlopen" ? Are they somehow determined ahead of time and kind of pre-linked? i.e. is your dlopen actually able to load stuff at runtime off the wire?

Off topic now: How about I found out more about componentize-py and how it works? How does this work relate to the pyodide project?

dicej · 2023-11-08T17:02:08Z

Off topic now: How about I found out more about componentize-py and how it works? How does this work relate to the pyodide project?

Sounds great -- here's a talk I did about componentize-py at WasmCon: https://wasmcon2023.sched.com/event/1PCMP/introducing-componentize-py-a-tool-for-packaging-python-apps-as-components-joel-dice-fermyon-technologies (video and slides both available on that page). Happy to do a Zoom call, chat on Discord, or whatever as well.

I've chatted with Hood Chatham in detail about Pyodide, and it's been an inspiration for a lot of what I've done with componentize-py. The main differences are WASI instead of Emscripten, no JS or browser required, and heavy use of the Component Model.

dicej · 2023-11-08T18:08:15Z

Out of interest when you talk about injecting a working version of "dlopen" at "composition time" is that just the "libdl" functions that get linked in, or are you also talking about the shared libraries that are opened by "dlopen" ? Are they somehow determined ahead of time and kind of pre-linked? i.e. is your dlopen actually able to load stuff at runtime off the wire?

The wasm-tools component link step (which componentize-py does automatically after searching the dependency tree for native extension libraries) bundles libdl.so, libc.so, libc++.so (if needed by one or more native extensions), libpython3.11.so, and all the native extension libraries into a component. Of those, only the native extensions are made available via dlopen (using the --dl-openable flag). That involves statically generating lookup tables which dlopen and dlsym use at runtime to convert each (library, symbol) name pair into a function pointer.

This implementation of dlopen cannot load libraries at runtime -- they all have to be accounted for when the component is created. That's why I sometimes call it "pseudo-dynamic linking". However, the Component Model does allow components to import modules, so the actual module used for a given native extension could be resolved immediately prior to instantiating and running the component if desired. True runtime linking is not in scope at the moment, but it's certainly something we'd like to support eventually.

sbc100 · 2023-11-08T18:15:52Z

Out of interest when you talk about injecting a working version of "dlopen" at "composition time" is that just the "libdl" functions that get linked in, or are you also talking about the shared libraries that are opened by "dlopen" ? Are they somehow determined ahead of time and kind of pre-linked? i.e. is your dlopen actually able to load stuff at runtime off the wire?

The wasm-tools component link step (which componentize-py does automatically after searching the dependency tree for native extension libraries) bundles libdl.so, libc.so, libc++.so (if needed by one or more native extensions), libpython3.11.so, and all the native extension libraries into a component. Of those, only the native extensions are made available via dlopen (using the --dl-openable flag). That involves statically generating lookup tables which dlopen and dlsym use at runtime to convert each (library, symbol) name pair into a function pointer.

This implementation of dlopen cannot load libraries at runtime -- they all have to be accounted for when the component is created. That's why I sometimes call it "pseudo-dynamic linking". However, the Component Model does allow components to import modules, so the actual module used for a given native extension could be resolved immediately prior to instantiating and running the component if desired. True runtime linking is not in scope at the moment, but it's certainly something we'd like to support eventually.

Ah, yes that is what I think I remembered about this use case. It seems like that "pseudo-dynamic linking" in this case could probably be optimized by doing that link statically use wasm-ld and building things as .a rather than .so? Since it seems like building things .so at all here has no advantage over .a? Maybe that can be done later as an optimization?

dicej · 2023-11-08T18:37:43Z

Ah, yes that is what I think I remembered about this use case. It seems like that "pseudo-dynamic linking" in this case could probably be optimized by doing that link statically use wasm-ld and building things as .a rather than .so? Since it seems like building things .so at all here has no advantage over .a? Maybe that can be done later as an optimization?

.so files have a number of advantages over .a files:

They can be published as part of packages to e.g. pypi.org using the same build steps and infrastructure that package maintainers are already using to publish e.g. Windows, Mac, and Linux builds.
The ability to import .so files into components allows you to deploy lightweight components which reference libc.so, libpython3.11.so, any native extension libraries, etc. rather than include them. E.g. a FaaS host needs only one copy of libpython3.11.so rather than 50 statically-linked copies for 50 apps. Likewise for lapack_lite.cpython-311-wasm32-wasi.so or any of the other dozen or so NumPy libraries.
They don't require a linker or any other native build tooling to use -- pip install componentize-py is all you need (although if someone were to package wasm-ld as a Python package maybe this wouldn't be a big deal).

sbc100 · 2023-11-08T18:45:04Z

Ah, yes that is what I think I remembered about this use case. It seems like that "pseudo-dynamic linking" in this case could probably be optimized by doing that link statically use wasm-ld and building things as .a rather than .so? Since it seems like building things .so at all here has no advantage over .a? Maybe that can be done later as an optimization?

.so files have a number of advantages over .a files:

They can be published as part of packages to e.g. pypi.org using the same build steps and infrastructure that package maintainers are already using to publish e.g. Windows, Mac, and Linux builds.

The ability to import .so files into components allows you to deploy lightweight components which reference libc.so, libpython3.11.so, any native extension libraries, etc. rather than include them. E.g. a FaaS host needs only one copy of libpython3.11.so rather than 50 statically-linked copies for 50 apps. Likewise for lapack_lite.cpython-311-wasm32-wasi.so or any of the other dozen or so NumPy libraries.

They don't require a linker or any other native build tooling to use -- pip install componentize-py is all you need (although if someone were to package wasm-ld as a Python package maybe this wouldn't be a big deal).

Good points! It still might be interesting as to measure how much using .so files costs though. i.e. how much perf is being left on the table, especially given things like LTO. I don't have a good intuition for how much it might be .. but I would guess is a fair bit.

tschneidereit · 2023-11-09T16:35:31Z

Hey @sbc100 👋🏻 I agree that for use cases where having .so files isn't necessary, .a files would be preferable, and would probably improve performance quite a bit, at least for very chatty interfaces.

The use cases we're currently looking at unfortunately fundamentally require .so files, so we don't have a choice here. Besides the requirements for installed toolchains, as Joel mentions being able to share libraries among a potentially huge number of different applications deployed on the same server is a key reason why static linking is unfortunately just not an option.

Given that, do you think we could get this approach in? If anyone in the future has use cases that lend themselves to static linking, we'd be happy to help facilitate that work, e.g. by providing reviews and support.

sbc100 · 2023-11-09T16:50:38Z

Hey @sbc100 👋🏻 I agree that for use cases where having .so files isn't necessary, .a files would be preferable, and would probably improve performance quite a bit, at least for very chatty interfaces.

The use cases we're currently looking at unfortunately fundamentally require .so files, so we don't have a choice here. Besides the requirements for installed toolchains, as Joel mentions being able to share libraries among a potentially huge number of different applications deployed on the same server is a key reason why static linking is unfortunately just not an option.

Given that, do you think we could get this approach in? If anyone in the future has use cases that lend themselves to static linking, we'd be happy to help facilitate that work, e.g. by providing reviews and support.

Sure thing, I just have couple more nits.

I think mostly agreed to go with above proposal to putting these stubs in libdl.so, so that still needs to happen.

libc-top-half/musl/src/misc/dl.c

Per review feedback, it's easier to simply replace libdl.so with a working implementation at runtime than it is to override a handful of symbols in libc. Note that I've both added libdl.so and replaced the empty libdl.a we were previously creating with one that contains the stubs. I'm thinking we might as well be consistent about what symbols the .so and the .a contain. Otherwise, e.g. the CPython build gets confused when the dlfcn.h says `dlopen` etc. exist but libdl.a is empty. Signed-off-by: Joel Dice <[email protected]>

dicej · 2023-11-09T18:42:06Z

@sbc100 thanks for the detailed feedback. I believe I've addressed everything.

BTW, I noticed that wasi-libc was already creating an empty libdl.a, which confused CPython's build system when building the statically-linked python.wasm, so I replaced the empty libdl.a with one that has the same stubs that libdl.so has. That seemed easier than trying to make CPython understand that dlopen and friends are available for the shared library build but not for the statically-linked build.

For WASI, we use flag values which match MacOS rather than musl. This gives `RTLD_LOCAL` a non-zero value, avoiding ambiguity and allowing us to defer the decision of whether `RTLD_LOCAL` or `RTLD_GLOBAL` should be the default when neither is specified. We also avoid declaring `dladdr`, `dlinfo`, and friends on WASI since they are neither supported nor stubbed at this time. Signed-off-by: Joel Dice <[email protected]>

sbc100 · 2023-11-13T15:36:31Z

libc-top-half/musl/include/dlfcn.h

+#define RTLD_LOCAL    0x4
+#define RTLD_GLOBAL   0x8
+#define RTLD_NOLOAD   0x10
+#define RTLD_NODELETE 0x80


Why re-define all of these just because you want to make RTLD_LOCAL non-zero?

We don't have to. I just figured we should try to consistently match some existing system that is already widely used, and MacOS was the first one I found that had a non-zero RTLD_LOCAL. I don't have any strong feelings about it, though. Would you prefer I change only RTLD_LOCAL?

I think we should always prefer the minimal set of changes from musl... unless there is a good reason not to.

I just pushed another update that uses the musl values except for RTLD_LOCAL.

This minimizes the divergence from upstream while still giving us the flexibility to choose a default value later. Signed-off-by: Joel Dice <[email protected]>

libc-top-half/musl/src/misc/dl.c

Signed-off-by: Joel Dice <[email protected]>

abrown

It looks to me like all the concerns are addressed; I was generally in favor of this from the beginning but it looks like the discussion improved some parts. Thanks @dicej for working through all of this and giving clear explanations! Since there seems to be consensus on the main idea, I'll go ahead and merge this and we can work out any details in future PRs.

cf. WebAssembly/wasi-libc#443

dicej force-pushed the dynamic-linking-dlopen-dlsym branch from 0236773 to 3a14878 Compare October 20, 2023 17:00

abrown reviewed Oct 25, 2023

View reviewed changes

yamt reviewed Nov 7, 2023

View reviewed changes

sbc100 reviewed Nov 9, 2023

View reviewed changes

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

sbc100 approved these changes Nov 9, 2023

View reviewed changes

sbc100 reviewed Nov 13, 2023

View reviewed changes

use musl's RTLD_* flags except for RTLD_LOCAL

3e11e7b

This minimizes the divergence from upstream while still giving us the flexibility to choose a default value later. Signed-off-by: Joel Dice <[email protected]>

yamt approved these changes Nov 14, 2023

View reviewed changes

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

libc-top-half/musl/src/misc/dl.c Outdated Show resolved Hide resolved

use NULL instead of 0 for null pointers

b4b59f2

Signed-off-by: Joel Dice <[email protected]>

abrown approved these changes Nov 15, 2023

View reviewed changes

abrown merged commit b85d655 into WebAssembly:main Nov 15, 2023
8 checks passed

yamt added a commit to yamt/toywasm that referenced this pull request Dec 13, 2023

examples/libdl: sync with the wasi-libc abi

54b3a35

cf. WebAssembly/wasi-libc#443

yamt mentioned this pull request Dec 13, 2023

examples/libdl: sync with the wasi-libc abi yamt/toywasm#128

Merged

yamt added a commit to yamt/toywasm that referenced this pull request Dec 15, 2023

examples/libdl: sync with the wasi-libc abi

8976641

cf. WebAssembly/wasi-libc#443

add stubs for dlopen, dlsym, etc. #443

add stubs for dlopen, dlsym, etc. #443

Conversation

dicej commented Oct 20, 2023

abrown left a comment

Choose a reason for hiding this comment

sbc100 commented Oct 26, 2023

dicej commented Oct 27, 2023 • edited Loading

sbc100 commented Oct 27, 2023

dicej commented Oct 27, 2023 • edited Loading

sbc100 commented Oct 27, 2023

sbc100 commented Oct 27, 2023 • edited Loading

dicej commented Oct 27, 2023

dicej commented Oct 27, 2023

dicej commented Oct 27, 2023

yamt commented Nov 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dicej Nov 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbc100 commented Nov 7, 2023

dicej commented Nov 8, 2023

yamt commented Nov 8, 2023

dicej commented Nov 8, 2023

sbc100 commented Nov 8, 2023

dicej commented Nov 8, 2023

sbc100 commented Nov 8, 2023

dicej commented Nov 8, 2023

sbc100 commented Nov 8, 2023

sbc100 commented Nov 8, 2023 • edited Loading

dicej commented Nov 8, 2023

dicej commented Nov 8, 2023

sbc100 commented Nov 8, 2023

dicej commented Nov 8, 2023 • edited Loading

sbc100 commented Nov 8, 2023

tschneidereit commented Nov 9, 2023

sbc100 commented Nov 9, 2023

dicej commented Nov 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrown left a comment

Choose a reason for hiding this comment

dicej commented Oct 27, 2023 •

edited

Loading

dicej commented Oct 27, 2023 •

edited

Loading

sbc100 commented Oct 27, 2023 •

edited

Loading

dicej Nov 10, 2023 •

edited

Loading

sbc100 commented Nov 8, 2023 •

edited

Loading

dicej commented Nov 8, 2023 •

edited

Loading

dicej commented Nov 9, 2023 •

edited

Loading