npm ci with esbuild on different systems

[schettgen]

Preconditions

We are using windows during development and linux docker for building.
node_modules folder is not committed...

When I now have esbuild as dependency in my project I install the deps and get

• esbuild
• @esbuild/win32-x64

Now package-lock.json is generated

Issue

We are now in linux docker container

We install there the deps with npm ci
Then we get the following message

#15 73.61 /src/Web/node_modules/esbuild/lib/main.js:1747
#15 73.61         throw new Error(`The package "${pkg}" could not be found, and is needed by esbuild.
#15 73.61               ^
#15 73.61 
#15 73.61 Error: The package "@esbuild/linux-x64" could not be found, and is needed by esbuild.

[hyrious]

npm ci with lockfile generated on different systems should NOT affect the installation of esbuild. Because the lockfile will include all optional dependencies' information and there's no difference in this file on different systems.

This is easy to verify since many projects on GitHub has actions running npm ci on different systems and Node.js versions to check compatibility.

Therefore, you should check if anything is different on Windows and on your linux docker container. For example the package-lock.json - lockfileVersion. Also you can try to install your project without the lockfile (or just run npm install on linux) and see if it generates anything different than your local setup.

[evanw]

That’s very unfortunate. It sounds like a bug with npm. I looked through npm’s issue tracker and found npm/cli#4828, which sounds like what you’re encountering. I wasn’t aware of this problem already. It sounds like it affects all tools that distribute native components (including swc and rollup, not just esbuild).

I’m not sure what to do about this. I read through the comments on that issue and it doesn’t seem like anyone from the npm team acknowledged the issue or indicated a willingness to fix it. Using another package manager without the bug may avoid this issue (e.g. Yarn has dedicated features for multi-platform packages).

But if npm itself (the canonical JavaScript package manager) has trouble with esbuild’s current package distribution strategy and is unlikely to fix their bugs, then esbuild may need a new package installation strategy. Or I somehow need to get npm fixed myself I suppose. Hmm…

[hyrious]

That's weird because I just tried that on my Windows machine and it generates correct lockfile…

So as far as I can tell from that npm cli issue, this problem occurs with:

• When it happens, package-lock.json only includes platform-specific packages. This is not happening on my machine.
• To trigger the issue, the target machine must have node_modules folder present. However as @schettgen said node_modules folder is not committed.

[schettgen]

Just for documentation purposes here my version

npm -v
10.7.0 

node -v
v20.14.0

To trigger the issue, the target machine must have node_modules folder present. However as @schettgen said node_modules folder is not committed.

Checked again no node_modules folder present

[schettgen]

Found a (dirty) workaround for now

npm ci --ignore-scripts
npm explore esbuild -- npm run postinstall

[jakebailey]

IIRC this npm bug only happens when there's already a lockfile; if you remove it and node_modules, then install again, you should see all of the missing deps appear.