SWIG JavaScript Evolution

This is SWIG JavaScript Evolution, an up-to-date fork of the SWIG project with a focus on JavaScript - native/Node.js, WASM/Node.js and WASM/browser.

Major New Features

• Interfacing with Node.js entirely through Node-API

  • Allowing for the distribution of universal pre-built binary modules, independent of the Node.js version, through npm - ie, your user types npm install and can start using your C++ library a few seconds later without having a build environment
  • worker_threads support out of the box
  • Automatic support for alternative Node.js derived frameworks such as Electron (This part of SWIG JSE has already been merged in the mainline SWIG 4.2.0)

• Automatic support for multi-threaded asynchronous execution for all C++ code through a simple flip switch One of the weakest points of Node.js in particular, and JavaScript in general, is its multithreading model. SWIG JSE changes this, bringing powerful multithreading support to developers coming from the Web world with no previous parallel programming experience. All locking is automatic and transparent and it includes deadlock prevention. Instead of writing:

  const result = nativeModule.mathHeavyStuff();

  your user writes:

  const result = await nativeModule.mathHeavyStuffAsync();

  and the code automatically runs in a separate background thread. Both CPU-heavy and I/O-heavy operations become automatically asynchronous. (This part of SWIG JSE is currently being considered for merging in the mainline SWIG 4.2.0)

• TypeScript Support

  Since its inception, TypeScript has surfed the JavaScript wave and after several iterations, it is becoming the language of choice for JavaScript development in the enterprise world - because of the additional coding discipline that it imposes. Modules produced by SWIG JSE support both JavaScript and TypeScript - and are fully typed when used from TypeScript - imposing the additional checks that are expected of TypeScript developers.

• WASM Support

  The true reason for the mass adoption of JavaScript is the fact that to this date, it remains the single language supported natively by web browsers. As more and more software moves to the web, there is a huge demand for porting legacy C/C++ software to this new platform. WASM is the answer to this demand, but porting C/C++ software to JavaScript also requires writing a huge amount of glue code - which is also the main object of SWIG. Thus, naturally, SWIG also evolves towards this new platform.

  Just like JavaScript offers the unique possibility of directly sharing code between the front-end (the browser) and the back-end (usually Node.js, but now there are alternatives), so SWIG JSE offers the unique possibility of writing an universal JavaScript wrapper, which can be compiled to a native Node-API addon for Node.js and to a WASM binary for the browser - with the two having identical APIs and sharing every single bit of code.

  A first project, magickwand.js has been published to npm as a testament to the capabilities offered by this new tool. It features the full ImageMagick-7 library - including support for multithreaded operations through async/await and TypeScript bindings - generated by about 700 lines of SWIG code that produces 400K lines of C++ code. Both the Node.js and the browser versions support async/await with an absolutely identical interface.

  Unlike all other supported languages, SWIG WASM does not target native code - on this platform JavaScript is the native language and the C/C++ code is the exotic environment. SWIG WASM is made possible by emnapi which provides a Node-API interface for interfacing with the JS engine in the browser.

• Code Splitting

  Although this feature has been mentioned many times in the past, the memory requirements of WASM compilation make it mandatory. SWIG JSE can produce separate C++ headers and multiple compilation units. At the moment only JavaScript Node-API and Javascript WASM can use this feature, but it is planned to eventually extend it to Python.

Usage

SWIG JavaScript Evolution is currently available only as a build from source from this repository and as a Github Action.

You can find an example skeleton for a new project that uses a dual-build Node.js/native and browser/WASM system at swig-napi-example-project.

If you want to see a real-world complex project that uses conan to manage a large pool of dependencies - including WASM builds - you should take a look at magickwand.js - the ImageMagick-7 bindings for JavaScript.

I am currently working on a second generation C++/JavaScript project with a new build system which will be based on the meson/conan/xpm trio instead of node-gyp - it will feature clean integration of 3rd party C++ libraries on all platforms (conan), use of the native build system (meson) and fully self-contained builds that do not need a working C++ environment on the final user host (xpm). This new project is the PROJ bindings for JavaScript. The project is partially functional and still not published.

Install

Building from source

# from git
git clone https://github.com/mmomtchev/swig.git

# from a release
tar -zxvf v5.0.0.tar.gz

cd swig

# add a suffix if you want to be able to use the main trunk SWIG
# on the same host
./configure --program-suffix=-jse

# this will install everything in /usr/local/share/swig-jse
# w/o affecting your existing SWIG
make -j 4 && sudo make install

For WASM, you will also need to install emsdk and setup its environment variables as described at https://github.com/emscripten-core/emsdk.

Conan recipes

SWIG JSE is now available on conan. You can simply add it to your conanfile.txt:

[tool_requires]
swig-jse/[>=5.0.3]
# optionally, get also emscripten
emsdk/[>=3.1.50]

It is available on my custom conan repository:

conan remote add swig-jse https://swig.momtchev.com/artifactory/api/conan/swig-jse

You can either use the prebuilt packages - which should now work on all three major OS - or - should you be unsettled by the recent events around xz-utils - you can also build it yourself.

The conan recipes are available at https://github.com/mmomtchev/swig-conan.

JavaScript manual

The only changes in the manual relative to the mainline SWIG are in the JavaScript section.

Maturity

Late technology preview / Early public release. The first open-source package - magickwand.js - is published on npm and it is used in production on some low-traffic websites. It uses all major new SWIG JSE features - async, WASM, locking and TypeScript. A second one, proj.js is currently in the works.

Planned major features

WASM without COOP/COEP

Currently, WASM projects using asynchronous wrappers require that COOP/COEP is enabled. In this example, it is enabled by the webpack built-in server and by the karma test runner. Users of your module will have to host it on web servers that support and send these headers - this is a requirement on the web server end - ie a configuration option that must be enabled in Apache or nginx. For example, currently Github Pages and many low-end hosting providers do not support it.

Alternatively, this example can be built without asynchronous wrappers in order to produce a WASM binary that does not require COOP/COEP. The only real difference is the emscripten build configuration which can be found in emscripten.gypi.

In this case, there are two possible strategies:

• Accept that calling C++ functions will produce main thread latency - which works well if all your C++ methods run very fast
• Use GoogleCromeLabs/comlink to call them in a worker thread - which works well if all your C++ methods have very long execution times because it adds significant overhead when calling them (this will require a custom serializer for C++ object - I plan to make an example)

Mixing the two is possible, but C++ functions running in the main thread and C++ functions running the in comlink worker won't be able to share objects as they will be running in separate memory spaces.

Build systems

SWIG-generated projects for JavaScript can currently choose between two build systems:

Description	node-gyp	meson + conan + xpm
Overview	The official Node.js and Node.js native addon build system from the Node.js core team	A new, still under development, experimental build system from SWIG JSE
Status	Very mature	Still not completely finished
Platforms with native builds	All platforms supported by Node.js	Linux, Windows and macOS
WASM builds	Hackish, see swig-napi-example-project and [email protected] for solutions	Out-of-the-box
Node.js APIs	All APIs, including the now obsolete raw V8 and NAN and the current Node-API	Only Node-API
Integration with other builds systems for external dependencies	Very hackish, see [email protected] for solutions, the only good solution is to recreate the build system of all dependencies around node-gyp	Out-of-the-box support for meson, CMake and autotools
conan integration	Very hackish, see [email protected]	Out-of-the-box
Build configurations through npm install CLI options	Yes	Yes
Distributing prebuilt binaries	Yes, multiple options, including @mapbox/node-pre-gyp, prebuild-install and prebuildify	prebuild-install
Requirements for the target host when installing from source	Node.js, Python and a working C++17 build environment	Only Node.js when using xpack-dev-tools, a working C++17 build environment otherwise
Makefile language	Obscure and obsolete (gyp)	Modern and supported (meson)

When choosing a build system, if your project:

• targets only Node.js/native and has no dependencies

  → stay on node-gyp

• meant to be distributed only as binaries compiled in a controlled environment

  → stay on node-gyp

• has a dual-environment native/WASM setup

  → node-gyp will work for you, but hadron has some advantages

• has dependencies with different build systems (meson, CMake, autotools)

  → hadron is the much better choice

• uses conan

  → hadron is the much better choice

• everything at once

  → hadron is the only choice

Project templates

You can find the classical SWIG Node-API project skeleton using node-gyp here. It has a dual-environment setup with Node.js/native and Browser/WASM builds.

You can find the new SWIG Node-API project skeleton using hadron here. It has a dual-environment setup with Node.js/native and Browser/WASM builds and external dependencies using conan.

conan / meson

Alas, the current state of my affair has made working with conan and meson extremely difficult.

In both projects, they tried to play a psychosis game with my work (my PRs) by doing simultaneously seemingly random acts and then trying to send me to see a psychiatrist. Given the context of the current affair, this has made any real work extremely difficult.

At the moment, both software packages need to be installed from my own repositories.