preprocess inlang.config.js to be able to use the advantages of an npm workflow

[ivanhofer]

To be honest, the current way to declare the inlang.config.js is not a great developer experience.

Problem

1. there is no typesafety
   If I want to add that, I would need to install the package via npm and add a type cast where I import it. But this only works for the packages we have released to npm. For others plugins like inlang-plugin-json this is not possible. Last week I needed to figure out how to upgrade from v1 to v2 and I needed to take a look at the implementation of the plugin and @inlang/core to know how I need to call the functions manually. TypeScript would have helped a lot there.
2. caching
   jsdelivr caches files for a full week. If we use majoir versioning in the docs (@1), people need to wait a full week to profit from improvements. If we would have a bug in a realease and jsdelivr fetches that version before we provide the fix, inlang will be broken for a full week.
   If we would use minor or patch versions in the docs, users would need to update the inlang config manually every few days to get the latest improvements.
3. users don't really know if there exists an update
   They would need to open the repository of each plugin to see if there is a new version.
   We could add a inlang update command to the CLI, but it is additional development effort and probably 90% of users will never use it because they don't know the option.
4. version mismatch.
   @inlang/sdk-js needs to be installed as an npm package but the plugin (to configure it) needs to be used using the jsdelivr link. So when you update the npm package, you will also need to remember to update the version in inlang.config.js or else you will probably have some strange behavior if the config changes and you are running the application next time.
5. developers don't know how "popular" a package is. They will probably take a look at npm to know how many weekly downloads it has. They will either not find it or if released just for the TypeScript definitions, they will see less downloads than it actually has.
6. not using "the platform"
   The whole JS ecosystem builds on npm (for better or worse). There are workflows (npm update), tooling (TypeScript) and knowledge (use import to load a module) that probably every developer knows. We should use that to our advantage and not introduce a new system. If we get native support for imports from urls in a few years, we can evaluate this again. (But we still would have some problems outlined in the points above)

Proposal

1. Let users write the config using regular npm imports.
2. Transform the file when reading it. npm imports will be transformed into jsdelivr fetch calls, using the version information from package.json
3. run that transformed file

@benjaminpreiss, master of ASTs, can help us with this task.

migration

We could write a migration script that automatically "upgrades" the config to the new system. This could be a cli command or we could just do it when e.g. the editor reads the file.

[felixhaeberle]

@ivanhofer I think you make valid points here. I wrapped my head around it for the last 10 minutes and according to "future-poofing" the inlang config, this would be a great enhancement.

I think it can be decided based on a draft implementation which performs better in the points above and doesn't miss anything what the config already has.

Maybe hard part:

• Make sure that the transformation can happen in all the places the config gets read, e.g. browser / server / CI+CD
• The migration has to be executed very good. Otherwise we will loose first users. But I think we can test this.

[samuelstroschein]

What I meant is: we could replicate the behavior that you get if you don't have a package.lock. If you set ^1.3.3 as your version then 1.5.7 could get installed if you run npm i the first time

I figured. For source-code stuff that makes sense. For plugins/extensions probably not so much. If I would have to update my vscode extension manually, I would go crazy :D

[ivanhofer]

Then the plugin should probably handle updating the version string automatically? So when the vite plugin runs, it will check the version string and update it. The same goes for the ide extension. But then each develper would need to make sure to have the same version installed ^^

[samuelstroschein]

So when the vite plugin runs, it will check the version string and update it.

We will have to introduce something in the future that goes beyond major versioning updating. But for now, plugins don't have access to vite/server/node and can't let another process auto-update the strings in a config.

[ivanhofer]

The sdk plugin can do that. It can update the contents of the inlang.config.js and restart the vite server. And I would assume the VS Code extension can also restart itself

[samuelstroschein]

The sdk plugin can do that. It can update the contents of the inlang.config.js and restart the vite server. And I would assume the VS Code extension can also restart itself

Had a similar idea in #669 (reply in thread). TL;DR anything that only happens in @inlang/sdk-js should be fine. Anything that requires changes/introduces node dependencies in @inlang/core will require lengthy discussions.

[NilsJacobsen]

So does it mean that we have a breaking change in the config file?

[ivanhofer]

Not really, it is just an improved way to write the config file. Similar to the plugin system we have introduced last week, this is optional and will not break existing configs.

[samuelstroschein]

I acknowledge that the DX of writing the config is improbable. But, I push back hard on introducing any NPM or node dependency:

• the DX issues are solvable by enhancing standard JS (node is not the platform)
• a dependency on NPM or node becomes redundant if ESM is enhanced, leading to long-term (redundant) maintenance effort and redundant complexity
• inlang is javascript independent. It just happens to be the case that inlang itself and plugins are written in JS (making an sdk-js easy). Any NPM dependency will break this theorem, making inlang for Flutter, iOS, Android, Mac, Windows apps less suitable.

Instead of "hacking" workarounds, we should be at the forefront of innovating JS to make JS better, our solution less complex, and ultimately deliver a better UX and DX.

Besides, the proposal of introducing transpilation will lead to noticeably longer loading times of the inlang editor. We are already battling loading times in the editor.

Replying to the mentions problems

1. there is no typesafety

We can and should (at some point) introduce type safety in multiple ways that do not depend on node or npm.

1. One possibility is extending TypeScript to support uri imports similar to Deno. Please read Typesafety for inlang.config.js || allow top level imports from node modules in `inlang.config.js` #431 (reply in thread).
2. Store types of imports in a .inlang directory and use JSDoc that resolves from a .inlang directory.

2. Caching

jsdelivr caches files for a full week. If we use majoir versioning in the docs (@1), people need to wait a full week to profit from improvements.

This is a non-issue. You can prune the cache https://www.jsdelivr.com/tools/purge :) On the flipside, NPM doesn't auto-update at all. That's what package.lock is for which makes sense for source code but less so for plugins.

3. users don't really know if there exists an update

We could add a inlang update command to the CLI, but it is additional development effort and probably 90% of users will never use it because they don't know the option.

This is exactly what we need to introduce. Companies will use the inlang editor, CLI and ide extension without the SDK-JS. How would we be able to notify users in the editor that a plugin update exists if we rely on NPM?

4. version mismatch.

@inlang/sdk-js needs to be installed as an npm package but the plugin (to configure it) needs to be used using the jsdelivr link. So when you update the npm package, you will also need to remember to update the version in inlang.config.js or else you will probably have some strange behavior if the config changes and you are running the application next time.

Given, this is an ugly edge case. But one that only exists because we also offer an SDK-JS (will only happen if the SDK-jS is used). Coming up with a working solution is desired. One way could be to let the inlang sdk plugin export a version that is read by the vite plugin and the vite plugin installs the correct node dependency.

1. sdkPlugin contains a version number
2. vite plugin has access to the version number by executing defineConfig
3. vite plugin install the correct NPM dependency / sets the package.json dependency

5. developers don't know how "popular" a package is.

developers don't know how "popular" a package is. They will probably take a look at npm to know how many weekly downloads it has. They will either not find it or if released just for the TypeScript definitions, they will see less downloads than it actually has.

We need a plugin registry at some point. That plugin registry will become the go-to directory, not npm. Ofc this plugin registry can include downloads. PS JSDelivr has download stats.

6. not using "the platform"

NPM and node are not the platform. ECMAScript and the browser is the platform.

There are workflows (npm update), tooling (TypeScript) and knowledge (use import to load a module) that probably every developer knows. We should use that to our advantage and not introduce a new system.

We have to introduce a system that is cross-environment compatible. The inlang environment exists to ensure that imports and access to the filesystem work in any environment, not to have a fancier import. The reality is that the inlang editor is cutting edge innovation. There exists no standard to import plugins and access the filesystem in the browser [that also works on the server]. We are the ones building that standard because we have to.

If we get native support for imports from urls in a few years, we can evaluate this again. (But we still would have some problems outlined in the points above)

Fun fact, url imports are long supported in ECMAScript/Browsers but not node. Node is introducing them though and this isn't even the main reason for $import. $import ensures that local JavaScript files in a repository can be imported in a browser environment.

Summary

The main point seems TypeSafety and yes that is a pain. If we have more users, revenue, and manpower, we should introduce typesafety for URI imports ❗

But, any NPM or node dependency will break inlang on multiple fronts. Typesafety itself is an example. How would a Flutter project get TypeSafety without NPM?

[ivanhofer]

I have picked statments from your post above and have shared my opinion:

• node is not the platform

For local develpment it currently is or why do we have a package.json inside our repository? ^^. We hopefully get to a cross-environment solution, but this will take some time.

inlang is javascript independent

I don't see this coming soon. Certainly not in 2023.

Besides, the proposal of introducing transpilation will lead to noticeably longer loading times of the inlang editor

I don't have any benchmarks but I expect it to take not more than 100ms, which is just a small part of the whole process.

You can prune the cache jsdelivr.com/tools/purge

That is new for me! Then we should certainly put this into our deployment scripts. #678

How would we be able to notify users in the editor that a plugin update exists if we rely on NPM?

Deleopers set up the config file and they decide which version schema they choose. And they will need to decide when they update it. By doing it like this: #669 (reply in thread) we have the exactly same behavior.

One way could be to let the inlang sdk plugin export a version that is read by the vite plugin and the vite plugin installs the correct node dependency

Please no! If developers decide to update a dependency they do it inside package.json and not inside a random config file.

We have to introduce a system that is cross-environment compatible

We are currently hacking it together by intorducing a new concept (environment functions) to make it work on all platforms. We already know the syntax how it will look like in the future and it is called import statements. So once all platforms support those, inlang will need to introduce a (not necessarily) breaking change to use the platform features.

There exists no standard to import plugins and access the filesystem in the browser

import is the standard

We are the ones building that standard because we have to.

The standard is arleady defined, just not available right now.

Summary

In almost all points you say: we would need to build tooling to support it.

If we would introduce the transfomration step we could already use everything almost for free. The only thing that would need to be built is the transformation process and from my experience it will probably take ~2-3 days. We could have it by next week. We could have a really great developer experience NOW and not a halb baked custom solution that needs to be explained to other developers (the perfect example just popped up: samuelstroschein/inlang-plugin-json#16 (comment)). Once all platforms are matured, we can get rid of the transformation process and nothing needs to be changed. If npm dies in a few years, we can just replace the import statements with an url instead of the package name.

[samuelstroschein]

node is not the platform
For local develpment it currently is or why do we have a package.json inside our repository? ^^. We hopefully get to a cross-environment solution, but this will take some time.

This is a unique edge case that effects the sdk-js. The rest of inlang is targeting standard JS that works across different environments.

inlang is javascript independent
I don't see this coming soon. Certainly not in 2023.

It's coming, latest in 2024. Several people expressed that they want to build Flutter plugins including @openscript. Once the web (not node) platform is nailed, Flutter will follow. I am already in conversations with agencies.

Besides, the proposal of introducing transpilation will lead to noticeably longer loading times of the inlang editor
I don't have any benchmarks but I expect it to take not more than 100ms, which is just a small part of the whole process.

The problem is mainly that most pain points can be resolved by using web standards, or extending web standards, rather than establishing a node dependency.

We have to introduce a system that is cross-environment compatible
We are currently hacking it together by intorducing a new concept (environment functions) to make it work on all platforms. We already know the syntax how it will look like in the future and it is called import statements. So once all platforms support those, inlang will need to introduce a (not necessarily) breaking change to use the platform features.

The key point is that the concept of environment functions (renamed to inlang environment btw) is plain JS. Nothing special is going on except that $import is a polyfill AND enables the browser to import local files.

We could have a really great developer experience NOW and not a halb baked custom solution that needs to be explained to other developers (the perfect example just popped up: samuelstroschein/inlang-plugin-json#16 (comment)). Once all platforms are matured, we can get rid of the transformation process and nothing needs to be changed.

I argue that introducing node dependencies degrades the UX more than it improves the overall DX of inlang.

Yes, the DX of the sdk-js would be better. But, the overall DX of inlang will be lower because a) Flutter, iOS, Android, Deno, Bun literally everyone that doesn't use node will suffer b) we have to maintain this, taking time from building things that matter c) it only affects the SDK and d) a solution with web standards can be developed solving a, b and c.

This will be an endless argument "we can improve DX in the short term if we use node". But, node dependencies are irrelevant to 3/4 of apps and literally every plugin of inlang.

One way could be to let the inlang sdk plugin export a version that is read by the vite plugin and the vite plugin installs the correct node dependency

Please no! If developers decide to update a dependency they do it inside package.json and not inside a random config file.

The only immediate action I see we need to take is the version mismatching. We definitely want one version to be defined. Given my stance on no node dependencies, I am okay if with anything that doesn't happen in @inlang/core.

For example, implementing a "version matching" process in @inlang/sdk-js/vite that updates either the inlang config or package.json based on which one is a higher version makes a ton of sense and wouldn't introduce node dependencies in @inlang/core.

If npm dies in a few years, we can just replace the import statements with an url instead of the package name.

I propose to read Designing Large Applications from the (now) CTO of Vercel.

He makes the point that things that will become redundant in the future should not be implemented because a) it becomes redundant and b) makes the application more complex.

I am sticking to the theorem of pushing back against edge-case improvements. Node dependencies are a no-go that we only need until we figure out a better way. A better way exists out there and I am sure we will design it. I will push hard against any node dependency as it makes inlang complex for a tiny subset of (potential) users. Again, anything in @inlang/sdk-js that does not require changes in @inlang/core should be fine.

[samuelstroschein]

Btw, @ivanhofer you are totally right. We want to use regular import calls. Import maps are coming https://github.com/WICG/import-maps. The concept of env.$import will (and should!) become redundant.

But, implementing import transpilation doesn't seem like the thing we should focus on right now because nailing and making import more bulletproof will be much harder than exposing $import.

[ivanhofer]

In the end my proposal is just a polyfill for funtionality that will come in the future. Once it is here, we can get rid of it. Without any breaking change. We don't require any users to use node dependencies. Import statements will also work for urls. We just greatly improve the DX by having standard tooling support that is established in the JS ecosystem.

I would assume that we don't have a better solution to this problem for over a year, so I think we will loose a lot of users who are stuck while setting inlang up or find the process to complicated. We will get asked every other week about why can't we use regular import statements. Even if it is mentioned in bold text every second line in the documentation, people won't get it.

[ivanhofer]

I had a call with @samuelstroschein today and we have found ways to improve 3 of the above described issues:

1. there is no typesafety: make $import typesafe #689
2. caching: purge jsdelivr caches when deploying  #678
3. version mismatch: fix version mismatch issue of @inlang/sdk-js and @inlang/sdk-js-plugin #691

Please take a look at the issues and add your thoughts there

[samuelstroschein]

We decided to make $import typesafe instead of polyfilling import for now because:

1. No transformation needed
2. Explicit “hey this is a special import”
3. Devs will otherwise try to import “node:fs/promises” etc. instead of using `env.$fs`.