Import memory from the other module with wasi

[anton-galkovsky]

Hello! I'm trying to share wasm memory between 2 instances. Could you help me? I've prepared an example that shows (for me) some strange behaviour.

Let's assume that we want to create 2 wasm files -- library.wasm (a bunch of useful functions/types/etc that we want to build once) and user_code.wasm (a few functions that will interact with the contents of the library that we want to rebuild multiple times). Functions from library.wasm and user_code.wasm should be able to pass pointers to some data or objects to each other and call their methods from each other, so we have to provide the same memory space for them.

Here is an example of library.cpp file:

#include <iostream>
#include <emscripten/emscripten.h>


extern "C" {

    EMSCRIPTEN_KEEPALIVE
    int new_int_ptr() {
        std::cout << "############# library new_int_ptr() called #############" << std::endl;
        int *x = new int(42);
        return (int) x;
    }

    int main() {
        std::cout << "############# library main() called #############" << std::endl;
        return 123;
    }
}

An example of user_code.cpp file:

#include <iostream>
#include <emscripten/emscripten.h>


extern "C" {

    EM_IMPORT(new_int_ptr)
    int new_int_ptr();

    EMSCRIPTEN_KEEPALIVE
    int get_new_int() {
        int x = new_int_ptr();
        return *(int *) x;
    }

    int main() {
        std::cout << "############# user code main() called #############" << std::endl;
        return 101;
    }
}

And main.go file:

package main

import (
	"fmt"
	"io"
	"os"
	"testing"

	"github.com/bytecodealliance/wasmtime-go/v14"
)

var t = &testing.T{}

func ReadWasmBytes(engine *wasmtime.Engine, path string) []byte {
	wasmFile, _ := os.Open(path)
	wasm, _ := io.ReadAll(wasmFile)
	wasmtime.ModuleValidate(engine, wasm)
	return wasm
}

func main() {
	engine := wasmtime.NewEngine()
	store := wasmtime.NewStore(engine)

	libraryWasmBytes := ReadWasmBytes(engine, "library.wasm")
	userCodeWasmBytes := ReadWasmBytes(engine, "user_code.wasm")

	libraryModule, _ := wasmtime.NewModule(engine, libraryWasmBytes)
	userCodeModule, _ := wasmtime.NewModule(engine, userCodeWasmBytes)

	wasiConfig := wasmtime.NewWasiConfig()
	wasiConfig.InheritArgv()
	wasiConfig.InheritEnv()
	wasiConfig.InheritStdin()
	wasiConfig.InheritStderr()
	wasiConfig.InheritStdout()
	store.SetWasi(wasiConfig)

	linker := wasmtime.NewLinker(engine)
	linker.DefineWasi()

	libraryInstance, _ := linker.Instantiate(store, libraryModule)

	libraryMain := libraryInstance.GetFunc(store, "_start")
	_, err := libraryMain.Call(store)
	fmt.Println("Returned error from library main():", err)

	linker.DefineInstance(store, "env", libraryInstance)

	linker.AllowShadowing(true)

	proc_exit := wasmtime.WrapFunc(store, func(a int32) {
		fmt.Println("@@@@@@@@@@@@@ proc_exit() called @@@@@@@@@@@@@")
	})
	linker.Define(store, "wasi_snapshot_preview1", "proc_exit", proc_exit)

	fd_write := wasmtime.WrapFunc(store, func(fd, iovsPtr, iovsLen, nwrittenPtr int32) int32 {
		fmt.Println("@@@@@@@@@@@@@ fd_write() called @@@@@@@@@@@@@")
		return 0
	})
	linker.Define(store, "wasi_snapshot_preview1", "fd_write", fd_write)

	userCodeInstance, _ := linker.Instantiate(store, userCodeModule)

	userCodeMain := userCodeInstance.GetFunc(store, "_start")
	_, err = userCodeMain.Call(store)
	fmt.Println("Returned error from user code main():", err)

	get_new_int := userCodeInstance.GetFunc(store, "get_new_int")
	ans, err := get_new_int.Call(store)
	fmt.Println("Returned answer and error from get_new_int():", ans, err)
}

And the commands that I use to run the code:

emcc library.cpp -o library.wasm -s "EXPORTED_RUNTIME_METHODS=['ccall']" -O2
emcc user_code.cpp -Wl,--import-memory -o user_code.wasm -s "EXPORTED_RUNTIME_METHODS=['ccall']" -O2
go run main.go

If I run these commands I'll get this output:

############# library main() called #############
Returned error from library main(): error while executing at wasm backtrace:
    0: 0x10bf - <unknown>!<wasm function 14>
    1:  0xf36 - <unknown>!<wasm function 10>

Caused by:
    Exited with i32 exit status 123
Returned error from user code main(): error while executing at wasm backtrace:
    0: 0x1ad70 - <unknown>!<wasm function 475>
    1: 0x4218 - <unknown>!<wasm function 45>
    2:  0x860 - <unknown>!<wasm function 8>
    3:  0xb47 - <unknown>!<wasm function 10>

Caused by:
    0: memory fault at wasm address 0xffffffff in linear memory of size 0x1000000
    1: wasm trap: out of bounds memory access
Returned answer and error from get_new_int(): <nil> error while executing at wasm backtrace:
    0: 0x10bf - <unknown>!<wasm function 14>
    1: 0x10b2 - <unknown>!<wasm function 12>
    2: 0x4f17 - <unknown>!<wasm function 117>
    3: 0xfc4c - <unknown>!<wasm function 279>
    4:  0xb5d - <unknown>!<wasm function 8>
    5:  0xb3c - <unknown>!<wasm function 9>

Caused by:
    Exited with i32 exit status 1

Let's look at it more thoroughly. The first line tells us that main() from library was successfully called. The next returned error Exited with i32 exit status 123 is okay too (as far as I understand). But the next error is not: memory fault at wasm address 0xffffffff in linear memory of size 0x1000000 -- it occurs when we try to call main() from the user code. We are not also getting here the log fd_write() called for some reason (despite we have changed wasi_snapshot_preview1.fd_write). And the next error tells us that get_new_int() was called unsuccessfully too (and also our implementation of wasi_snapshot_preview1.proc_exit was not called).

But if we comment out std::cout << "############# user code main() called #############" << std::endl; line from the user code, the output will change a lot:

############# library main() called #############
Returned error from library main(): error while executing at wasm backtrace:
    0: 0x10bf - <unknown>!<wasm function 14>
    1:  0xf36 - <unknown>!<wasm function 10>

Caused by:
    Exited with i32 exit status 123
@@@@@@@@@@@@@ proc_exit() called @@@@@@@@@@@@@
Returned error from user code main(): error while executing at wasm backtrace:
    0:   0xfa - <unknown>!<wasm function 4>

Caused by:
    wasm trap: wasm `unreachable` instruction executed
############# library new_int_ptr() called #############
Returned answer and error from get_new_int(): 42 <nil>

Now all the functions have worked as expected: main() from user code was finished with proc_exit() called message (the `unreachable` instruction executed message is expected as far as I understand), the function new_int_ptr() was called successfully and we have observed that the instances use the same memory -- the number 42 was created on the library side and then read on the user code side by its pointer.

By changing this example a bit it is possible to face other errors, so I guess there is some kind of undefined behaviour in this example, but I don't know what causes the problem. Can you help me, please?

[alexcrichton]

Thanks for the report, but sorry there's quite a lot going on here and I'm not sure how much I'll be able to help. You're using emcc which I'm not familiar with (with respect to its non-browser use cases). In that sense it seems like you're running into behaviors of the C standard library compiled to wasm, and so that all seems emcc-specific.

Put another way, I suspect that much of this behavior has to do with emcc conventions which I don't know anything about myself, so in the absence of a known bug in these bindings I'm not sure how to help.

[anton-galkovsky]

Okay, thank you anyway!