Any chance to provide opencv ?

[deem0n]

Hi dicej!

Many thanks for wasi builds! We are using numpy from this repo, but also we need opencv.

My pathetic attempt to compile opencv resulted in error:

  Successfully installed cmake-3.30.5 distro-1.9.0 numpy-2.1.2 packaging-24.1 pip-24.3.1 scikit-build-0.18.1 setuptools-75.3.0 wheel-0.44.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/commands/wheel.py", line 147, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 545, in collect_root_requirements
    reqs = list(
           ^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 501, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 233, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in __init__
    super().__init__(
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 159, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 236, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 315, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 56, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 126, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 103, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 701, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 27, in <module>
    from .dist import Distribution
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 30, in <module>
    from . import (
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/command/__init__.py", line 9, in <module>
    from distutils.command.bdist import bdist
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/bdist.py", line 36, in <module>
    class bdist(Command):
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/bdist.py", line 45, in bdist
    f"[default: {get_platform()}]",
                 ^^^^^^^^^^^^^^
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/_distutils/util.py", line 60, in get_platform
    return get_host_platform()
           ^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/pip-build-env-038gzty6/overlay/lib/python3.12/site-packages/setuptools/_distutils/util.py", line 47, in get_host_platform
    return sysconfig.get_platform()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/sysconfig.py", line 831, in get_platform
    get_config_vars(),
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/sysconfig.py", line 736, in get_config_vars
    _init_config_vars()
  File "/usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/sysconfig.py", line 683, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/sysconfig.py", line 549, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata__wasi_wasm32-wasi'

Could you please help with this?

[dicej]

Hi @deem0n. This looks like it might be due to bytecodealliance/componentize-py#23. Specifically, the sysconfig crate seems to be loading the _sysconfigdata__wasi_wasm32-wasi module at runtime, but due to that issue componentize-py doesn't have access to that module at runtime -- only at build time. To force it to be loaded at build time, try adding a import _sysconfigdata__wasi_wasm32-wasi statement to the top level of your main application .py file.

See also https://github.com/bytecodealliance/componentize-py?tab=readme-ov-file#known-limitations and bytecodealliance/componentize-py#91

EDIT: Nevermind, looks like this is happening when building the wheel -- before componentize-py is part of the picture.

[deem0n]

basically I made submodule from https://github.com/opencv/opencv-python.git and using this build.sh

#!/bin/bash

set -eou pipefail

if [ ! -e venv ]; then
  python3.12 -m venv venv
fi

. venv/bin/activate
pip install --upgrade pip
pip install build wheel cython setuptools scikit-build

ARCH_TRIPLET=_wasi_wasm32-wasi

export CC="${WASI_SDK_PATH}/bin/clang"
export CXX="${WASI_SDK_PATH}/bin/clang++"

export PYTHONPATH=$CROSS_PREFIX/lib/python3.12

export CFLAGS="-I${CROSS_PREFIX}/include/python3.12 -D__EMSCRIPTEN__=1"
export CXXFLAGS="-I${CROSS_PREFIX}/include/python3.12"
export LDSHARED=${CC}
export AR="${WASI_SDK_PATH}/bin/ar"
export RANLIB=true
export LDFLAGS="-shared"
export _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_${ARCH_TRIPLET}

export ENABLE_HEADLESS=1
cd src
#python3 -m build -n -w
pip3 wheel . --verbose
wheel unpack --dest build dist/opencv*.whl

[dicej]

What is CROSS_PREFIX set to when you run that script? Is it pointing to a WASI built of PYTHON like in https://github.com/dicej/wasi-wheels/blob/main/Makefile ?

[deem0n]

I run it using main Makefile with

make /Users/deem0n/Documents/Projects/wasi-wheels/build/opencv-wasi.tar.gz

I printed out all env vars, CROSS_PREFIX=/Users/deem0n/Documents/Projects/wasi-wheels/cpython/builddir/wasi/install

AR=/Users/deem0n/Documents/Projects/wasi-wheels/build/wasi-sdk/bin/ar
ARCH_TRIPLET=_wasi_wasm32-wasi
BASH=/usr/local/bin/bash
CC=/Users/deem0n/Documents/Projects/wasi-wheels/build/wasi-sdk/bin/clang
CFLAGS='-I/Users/deem0n/Documents/Projects/wasi-wheels/cpython/builddir/wasi/install/include/python3.12 -D__EMSCRIPTEN__=1'
CROSS_PREFIX=/Users/deem0n/Documents/Projects/wasi-wheels/cpython/builddir/wasi/install
CXX=/Users/deem0n/Documents/Projects/wasi-wheels/build/wasi-sdk/bin/clang++
CXXFLAGS=-I/Users/deem0n/Documents/Projects/wasi-wheels/cpython/builddir/wasi/install/include/python3.12
ENABLE_HEADLESS=1
HOME=/Users/*****
LANG=ru_RU.UTF-8
LC_TERMINAL=iTerm2
LC_TERMINAL_VERSION=3.5.6beta1
LDFLAGS=-shared
LDSHARED=/Users/deem0n/Documents/Projects/wasi-wheels/build/wasi-sdk/bin/clang
MACHTYPE=x86_64-apple-darwin23.4.0
MAKEFLAGS=
MAKELEVEL=1
MFLAGS=
OLDPWD=/Users/deem0n/Documents/Projects/wasi-wheels/opencv
OPTERR=1
OPTIND=1
OSTYPE=darwin23.4.0
PAGER=less
PIPESTATUS=([0]="0")
PPID=87177
PS1='(venv) '
PS4='+ '
PWD=/Users/deem0n/Documents/Projects/wasi-wheels/opencv/src
PYTHONPATH=/Users/deem0n/Documents/Projects/wasi-wheels/cpython/builddir/wasi/install/lib/python3.12
RANLIB=true
SHELL=/bin/zsh
TMPDIR=/var/folders/jn/q10st3jd7ngb24nckb0cv49r0000gp/T/
UID=502
USER=deem0n
VIRTUAL_ENV=/Users/deem0n/Documents/Projects/wasi-wheels/opencv/venv
VIRTUAL_ENV_PROMPT='(venv) '
WASI_SDK_PATH=/Users/deem0n/Documents/Projects/wasi-wheels/build/wasi-sdk
WASMTIME_HOME=/Users/deem0n/.wasmtime
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
ZSH=/Users/deem0n/.oh-my-zsh
_=src
_OLD_VIRTUAL_PATH='/usr/local/sbin:/Users/deem0n/.wasmtime/bin:/Users/deem0n/.fvm/bin:/Users/deem0n/.fluvio/bin:/Users/deem0n/.cargo/bin:/Users/deem0n/.local/bin:/Users/deem0n/.orbstack/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/Applications/Wireshark.app/Contents/MacOS:/Library/Frameworks/EdgeDB.framework/bin:/usr/local/go/bin:/Users/deem0n/.cargo/bin:/Applications/iTerm 3.app/Contents/Resources/utilities'
_OLD_VIRTUAL_PS1=
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi

[dicej]

Thanks for the info. It's interesting that /usr/local/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/sysconfig.py is showing up in the stack trace, which indicates that the MacOS version of Python is being used rather than the WASI build; the latter will have the _sysconfigdata__wasi_wasm32-wasi module, but the former presumably won't. It seems that pip is using the former (which kind of makes sense because the WASI build of Python won't run natively, so it has to use the MacOS version), but then it's getting confused because it can't find the _sysconfigdata__wasi_wasm32-wasi module indicated by _PYTHON_SYSCONFIGDATA_NAME.

You might want to ask the OpenCV maintainers whether cross-compiling this package is supported (and if so, how). Not all Python packages support cross-compilation, and adding such a feature could be non-trivial.

Sorry I'm not more helpful; my Python skills is still pretty rudimentary, and I relied on @kesmit13 to show me how to cross-compile NumPy. @brettcannon might have ideas.

[deem0n]

Thanks Joel!
It's enough info to continue. I am also new to Python, but have some proper guidelines now!

[brettcannon]

Easiest way to work around this sort of import failure is to set the PYTHONPATH environment variable to point at the directory that contains _sysconfigdata__wasi_wasm32-wasi in your WASI build of CPython.