💭 Bundler & Plugins

[jer3m01]

Used to track discussion related to the rome bundler and the design of it's plugin system.

[jer3m01]

Plugins could be made using callbacks similar to Laravel's middleware https://laravel.com/docs/8.x/middleware#defining-middleware.

type data = {};
type next = (data: data) => data;
type config = {};

// Example struc
abstract class RomePlugin {
    // Whatever the name of the stages would be
    abstract compile(data: data, next: next, config: config): data;
    abstract bundle(data: data, next: next, config: config): data;
}

class pluginA extends RomePlugin {

    compile(data: data, next: next, config: config): data {
        data.doStuff();
        // Returning `data` instead of `next` would not run the other plugins
        return next(data);
    }

    bundle(data: data, next: next, config: config): data {
        // Using the return value of the closure would run all the other plugins before performing it's actions
        let processedData = next(data);

        processedData.doOtherStuff();

        return processedData;
    }
}

class pluginB extends RomePlugin {
    ...
}

// config plugin list, would run in order for each stage
plugins: [
    pluginB,
    pluginA,
]

This would also make the backend implementation fairly easy. This should in theory be working

const config;

// Repeat for each stage
function runCompile() {
    // Internal compiling code
    const internalData = internalFunction();

    let i = 0;
    const next = (data: data): data => {
        if (i >= config.plugins.length) return data;

        const plugin = config.plugins[i++];
        return new plugin().compile(data, next, config);
    }

    return next(internalData); // Fully compiled code
}

[ematipico]

I think the philosophy of the bundler should reflect what the modern web development needs.

We need a bundler for when we ship code to production (still) but we don't need a bundler for develop our applications locally. That's where snowpack and vite shine. I believe Rome should follow the same philosophy.

ESM modules for web development, optimized build for production.

That being said, about the plugin system, I think the concept of "plugin" is too broad and it's really difficult to understand what a plugin does. I like Parcel's approach, where they have specific naming convention that helps to understand the responsibility of a "plugin". They have transformers, optimizers, packagers, etc. and they are divided by file extension.

What do you think?

[jer3m01]

ESM modules for development and optimized builds for prod sound like the way to go.

Had a quick look to Parcel's plugin system, really like the idea of auto loading & splitting them.
The splitting is a bit similar to my example with the different stages, now how to implement it would need a bit more input.

@sebmck any takes on this?

[jer3m01]

Also should have a look at rollup's plugin system, heard positive feedback about them.

Some resources:
https://www.elliotclyde.nz/blog/rollup-plugins-for-beginners/

[jer3m01]

When bundling rome checks all files included in the bundle, we should maybe disable some checks for node_modules packages.
Causes problems such as searching for missing packages that are optionally required() by the lib.

[sebmck]

@jer3m01 We actually already have an optional field that we tag on dependencies when analyzing files. Right now I think it only checks for require() inside of a try-catch. When this happens we allow module resolution to fail. Is that what you had in mind?

[jer3m01]

Yes, didn't know we had that.
We should also add it to conditional require() i.e. if (cond) require() as this currently crashes.

[ajkachnic]

Are there any plans to make sure bundling is performant? It seems like WASM powered tools like esbuild are able to greatly outperform more conventional bundlers. I assume the general philosophy of no external dependencies still applies here, so that means a tool like esbuild is out of the question, and probably means any bundler would be written in TypeScript.

Just curious what kind of optimizations and performance tweaks would be needed to make this relatively fast

[yassere]

Here are some of my personal thoughts on plugins. Some of these are longer-term goals but would inform earlier decisions if we decided to pursue them.

A typical user should be able to use plugins without editing (or even reading) any configuration files. We should have a powerful rome plugin command that lets people (and build servers) add/enable/disable/discover/describe/configure plugins entirely from the command line. The rome cli should be able to provide structured data on which plugins are enabled, what capabilities they contribute, and how they’re configured. Reading/editing the config files manually should always be possible, of course. But I think it’s fine if the config format ends up being a bit verbose.

Plugins can contribute multiple different capabilities (lint rules, compiler transforms, etc) and they declare that in a manifest. Some sort of rome plugin describe command can tell you what a plugin claims to contribute and can check to make sure the plugin actually exports what it says it does. Possibly the cli can do some basic runtime validation of the exports. We could do those same checks any time a plugin is being added, enabled, or upgraded so that we can report diagnostics when something looks wrong.

Plugins aren’t required to be dedicated packages. If you maintain a package and you want to create some related rome lint rules, it should be possible to just add those capabilities to your existing package and tell your users what rome plugin command to run in order to enable them.

No naming conventions, not even a rome-plugin prefix. Plugins should declare their capabilities; they don’t need to be named after them. If we want to assist with discoverability, we can host some sort of directory.

[jamiebuilds]

No naming conventions, not even a rome-plugin prefix.

I disagree with this one, discovery is never going to be from one location so a directory does little to help. The other day I was trying to see what typescript plugins exist, but there's no naming scheme and it makes them very hard to find.

[jer3m01]

I disagree with this one, discovery is never going to be from one location so a directory does little to help. The other day I was trying to see what typescript plugins exist, but there's no naming scheme and it makes them very hard to find.

I agree with removing naming conventions but we do need a way to discover them easily and managing a dictionary would just be more work.
Using package.json keywords could be a good alternative, making discovery easy without restricting package names. i.e. rome-plugin

[yassere]

I can see how it makes sense to encourage plugin authors to name their npm packages using the rome-plugin prefix, but my personal preference is that Rome won't have any magic config/resolution based on it. I think we should have an explicit config format that most people won't have to deal with manually.

[jamiebuilds]

Oh yeah, I hate the name -> babel-plugin-name thing that we did in Babel configs. We got rid of that when designing the Parcel config format. But Parcel also forces a naming scheme and will emit annoying warnings if you don't follow it. We also added some recommendations on how to correctly name plugins: https://v2.parceljs.org/plugin-system/authoring-plugins/#naming

[ematipico]

I think we can't escape the naming discovery, especially if we think that rome config file won't be a .js file. So something like this is unavoidable:

name = "my-project"
plugins = [
	"@companyscope/fancy-rome-plugin",
	"foo-plugin-rome"
]

No restrictions on the name, but this depends on how a plugin will be. If we follow @sebmck's suggestion what we need is just a index.js/index.ts that exports what Rome needs. And we can also do validation checks before registering a plugin.