workers on yarn install in nodeLinker: node-modules mode

[sod]

Testing deno install cached installation:

yarn install --mode=skip-build  6,45s user 7,74s system 255% cpu 5,559 total
deno install  1,01s user 15,43s system 662% cpu 2,482 total

(comparing skip-build against deno, as deno ignores all post-install scripts by default)

My node_modules folder is ~500 MB in size. A typical angular application (this one https://github.com/sod/sod-monorepo).

Not sure if it's just rust or that they distribute the work over all cpu cores.

Do you think using workers to distribute creating the file system entries could greatly boost cached installs?

I'm sorry if I sound ridiculous as I'm aware that yarn solved this already with pnp mode. I just never switched, as it took too much time fighting the node ecosystem.

[larixer]

Do you think using workers to distribute creating the file system entries could greatly boost cached installs?

It might very well be. For what it worth I have tried to rewrite tar unpacking in Zig and saw 2-3x time boost in install times for my side project:
https://github.com/narijs/nari/tree/larixer/zig-extractor
which strangely coincides with figures boost you see :)

So, something is not very optimal with Node.js trying to write a bunch of files to the file system during install phase. It needs further investigation, exactly which operations in Node.js are inefficient and why. Perhaps just file writes?

[merceyz]

Do you think using workers to distribute creating the file system entries could greatly boost cached installs?

We're using async I/O so they probably wouldn't, to quote the docs:

Workers (threads) are useful for performing CPU-intensive JavaScript operations. They do not help much with I/O-intensive work. The Node.js built-in asynchronous I/O operations are more efficient than Workers can be.
https://nodejs.org/docs/v20.18.0/api/worker_threads.html#worker-threads

[sod]

Hm seem like your are right. If I bluntly inject myself by replacing all 'fs' with './fs_middleware.js' in yarn, with fs_middleware.js being:

const realfs = require('node:fs')
const { performance } = require('node:perf_hooks');

const fs = {};
const nonMeasuredCalls = {};
const measuredCalls = {};
const measurementSum = {};
const lastCalls = [];
let ioActive = 0;
let maxIoActive = 0;

for (const name of Object.keys(realfs)) {
    if (typeof realfs[name] !== 'function') {
        fs[name] = realfs[name];
        continue;
    }

    fs[name] = function() {
        const args = Array.from(arguments);
        const callback = args.at(-1);
        const id = name;

        lastCalls.length = 4;
        lastCalls.unshift(id);
        ioActive += 1;
        maxIoActive = Math.max(ioActive, maxIoActive);

        if (name.endsWith('Sync')) {
            const start = performance.now();
            const result = realfs[name].apply(this, arguments);
            const end = performance.now();
            measuredCalls[id] = (measuredCalls[id] ?? 0) + 1;
            measurementSum[id] = (measurementSum[id] ?? 0) + end - start;
            ioActive -= 1;
            return result;
        }

        if (typeof callback === 'function') {
            const start = performance.now();
            return realfs[name].apply(this, args.concat(function() {
                const end = performance.now();
                measuredCalls[id] = (measuredCalls[id] ?? 0) + 1;
                measurementSum[id] = (measurementSum[id] ?? 0) + end - start;
                ioActive -= 1;
                callback.apply(this, arguments);
            }))
        }

        const result = realfs[name].apply(this, arguments);
        nonMeasuredCalls[id] = (nonMeasuredCalls[id] ?? 0) + 1;
        ioActive -= 1;

        return result;
    }
}

process.on('exit', () => {
    console.log({nonMeasuredCalls, measuredCalls, measurementSum: Object.entries(measurementSum).map(([name, time]) => [name, time.toFixed(1) + 'ms']), lastCalls, maxIoActive})
})

module.exports = fs;

Then the result is:

{
  nonMeasuredCalls: {},
  measuredCalls: {
    existsSync: 117,
    readFileSync: 1144,
    mkdirSync: 1,
    writeFileSync: 1,
    lstatSync: 1143,
    symlink: 108,
    writeFile: 38024
  },
  measurementSum: [
    [ 'existsSync', '0.5ms' ],
    [ 'readFileSync', '42.1ms' ],
    [ 'mkdirSync', '0.0ms' ],
    [ 'writeFileSync', '0.2ms' ],
    [ 'lstatSync', '6.9ms' ],
    [ 'symlink', '5.9ms' ],
    [ 'writeFile', '7584.7ms' ]
  ],
  lastCalls: [ 'writeFile', 'mkdir', 'rmdir', 'lstat', 'lstat' ],
  maxIoActive: 22020
}

So yarn tries already to be as parallel as possible, but that darn writeFile is so slow.