Skip to content

Use multiple CPU cores via clusters for the fastest way to read nested directories and their stats

License

Notifications You must be signed in to change notification settings

bevry-archive/readdir-cluster

Repository files navigation

readdir-cluster

Status of the GitHub Workflow: bevry NPM version NPM downloads
GitHub Sponsors donate button ThanksDev donate button Patreon donate button Liberapay donate button Buy Me A Coffee donate button Open Collective donate button crypto donate button PayPal donate button
Discord server badge Twitch community badge

Create a cluster of workers to iterate through the filesystem

Usage

Complete API Documentation.

import readdirCluster, { Stat } from 'readdir-cluster'

// note that Stat is not the same as fs.Stats as it has functions removed, as it needed to be serialisable
function iterator(path: string, filename: string, stat: Stat) {
    // skip directories and files that start with .
    if (filename[0] === '.') return false
    // do not recurse into directories
    if (stat.directory) return false
}

const paths = await readdirCluster({ directory: '.', iterator })
console.log(paths)

Performance

Benchmarks:

  • Running readdir-cluster . returns 7388 files in 500ms

  • Running readdir with recursive: true returns 7388 files in 100ms

    import { readdir } from 'fs'
    readdir('.', { recursive: true }, (err, files) => {
        if (err) console.error(err)
        else if (files.length) process.stdout.write(files.join('\n') + '\n')
    })
  • Running fdir returns 6480 files in 100ms

    import { fdir } from 'fdir'
    const api = new fdir().withBasePath().crawl(process.argv[2])
    api.withPromise().then((files) => {
        if (files.length) process.stdout.write(files.join('\n') + '\n')
    })

Recommendations:

  • if you target Node.js 18.7 and above, you should use fs.readdir with recursive: true
  • if you target older Node.js versions, you should use @bevry/fs-list
  • if you target older Node.js versions and you want a stat object, use readdir-cluster
  • if you target Nodejs 12 and above, and want a lot of customisation, use fdir

As for why this package exists, readdir-cluster was created in 2005, recursive was added to Node.js in 2023, and fdir was created in 2020. That said, there are several issues that could potentially improve readdir-cluster performance.

Install

Install Globally

  • Install: npm install --global readdir-cluster
  • Executable: readdir-cluster

Install Locally

  • Install: npm install --save readdir-cluster
  • Executable: npx readdir-cluster
  • Import: import pkg from ('readdir-cluster')
  • Require: const pkg = require('readdir-cluster').default

This package is published with the following editions:

  • readdir-cluster aliases readdir-cluster/index.cjs which uses the Editions Autoloader to automatically select the correct edition for the consumer's environment
  • readdir-cluster/source/index.ts is TypeScript source code with Import for modules
  • readdir-cluster/edition-es2022/index.js is TypeScript compiled against ES2022 for Node.js 14 || 16 || 18 || 20 || 21 with Require for modules
  • readdir-cluster/edition-es2017/index.js is TypeScript compiled against ES2017 for Node.js 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with Require for modules
  • readdir-cluster/edition-es2015/index.js is TypeScript compiled against ES2015 for Node.js 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with Require for modules
  • readdir-cluster/edition-es5/index.js is TypeScript compiled against ES5 for Node.js 4 || 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with Require for modules
  • readdir-cluster/edition-es2017-esm/index.js is TypeScript compiled against ES2017 for Node.js 12 || 14 || 16 || 18 || 20 || 21 with Import for modules
  • readdir-cluster/edition-types/index.d.ts is TypeScript compiled Types with Import for modules

History

Discover the release history by heading on over to the HISTORY.md file.

Backers

Code

Discover how to contribute via the CONTRIBUTING.md file.

Authors

Maintainers

Contributors

Finances

GitHub Sponsors donate button ThanksDev donate button Patreon donate button Liberapay donate button Buy Me A Coffee donate button Open Collective donate button crypto donate button PayPal donate button

Sponsors

  • Andrew Nesbitt — Software engineer and researcher
  • Balsa — We're Balsa, and we're building tools for builders.
  • Codecov — Empower developers with tools to improve code quality and testing.
  • Poonacha Medappa
  • Rob Morris
  • Sentry — Real-time crash reporting for your web apps, mobile apps, and games.
  • Syntax — Syntax Podcast

Donors

License

Unless stated otherwise all works are:

and licensed under: