Enhance $fs interface

[openscript]

Currently the whole interface from memfs is used for $fs, while only using readFile and writeFile inside Inlangs core.

https://github.com/inlang/inlang/blob/bc7be04138aa0614a5686e59a8ef1497f9278edb/source-code/core/src/config/environment-functions/%24fs.ts#L1-L9

The type $fs is not used internally:
https://github.com/inlang/inlang/blob/bc7be04138aa0614a5686e59a8ef1497f9278edb/source-code/core/src/config/environment-functions/%24import.ts#L36
https://github.com/inlang/inlang/blob/bc7be04138aa0614a5686e59a8ef1497f9278edb/source-code/core/src/config/environment-functions/%24import.ts#L59

The type $fs is used externally for plugins. E. g.:

https://github.com/samuelstroschein/inlang-plugin-json/blob/c4fee39b226806863ad05a0d88fd618154848d86/src/index.ts#L40

Problem

From the ide-extension (VSCode) we also access the projects file system. Unfortunately using node:fs, which is mostly compatible with memfs, can have side-effects on other extensions. VSCode offers its own file system vscode.workspace.fs, which is not compatible with memfs and node:fs.

Proposal

• Define the minimal $fs (let's call it inlangfs) instead of using the whole memfs for Inlangs core.
  • Qualities:
    • async
  • Methods:
    • readFile
    • writeFile
    • readdir (vscode FileSystem: readDirectory)
• The interface for the functions (writeFile, readFile, ...) need to be compatible with the browsers, hence memfs.
• Use inlangfs everywhere in the core and plugins for type safety.
• Write a proxy mapper between inlangfs and vscode.workspace.fs for the ide-extension.

[samuelstroschein]

This discussion emerged on Discord (link):

@openscript:
I've started this discussion yesterday: microsoft/vscode-discussions#466 (reply in thread)

Now I've pulled VSCode and started analyzing the inlang extension inside a self built VS Code instance. He seems to be right. The extensions share a process. Some extensions start other processes (like the tsserver binary), which are children of the extensionHost process.

There is a proxy between the extensionHost and the extensions. For example the proxy prevents calling process.exit. When we process.chdir() for other extensions the working directory is changed too. This is bad. You were right Samuel.

To move forward we should use vscode.workspace.fs. There are a few steps to achieve this. First is to extend our own type/interface $fs, so it doesn't just use memfs but only the minimal fs inlang needs (currently readFile and writeFile).

Unfortunately vscode.workspace.fss readFile and writeFile are not compatible with the ones from memfs or node:fs. So we need to have a proxy mapper inside the extension to map fs calls from the core to the vscode.workspace.fs.

[samuelstroschein]

First is to extend our own type/interface $fs, so it doesn't just use memfs but only the minimal fs inlang needs (currently readFile and writeFile).

I agree. Several questions:

1. Do we need to support the minimum filesystem API the current git implementation (isomorphic-git) needs, or do we leave that up to the "runtime"?

The VSCode extension, for example, does not use the git implementation. Hence, filesystem APIs that are required for the "browser runtime" (that must use the git implementation) is not required for the IDE extension.

2. How do we solve the root problem of "different working directories"?

The root problem is relative paths that are resolved differently from the host (editor/vscode extension) and inlang.config.js. How do we ensure that the host provides an $fs implementation that treats the path of the inlang.config.js file as "working directory"?

[openscript]

1. I would define the minimum filesystem API for what the core needs. The editor and other users of the core can have an extended fs for their own needs.
2. In my opinion each inlang.config.js is a config instance. One config instance has its filesystem instance. The root of this filesystem is where the inlang.config.js resides. The ide-extension and editor have to manage these config instances, as they support multiple workspaces, hence multiple configs. "This" implementation has already started inside the ide-extension as it searches for the closest configuration instance.

[samuelstroschein]

1. Agree

Regarding 2:

The root of this filesystem is where the inlang.config.js resides [...] the ide-extension as it searches for the closest configuration instance.

That's how it's supposed to be, but isn't the problem that $fs calls need to resolve from that path?

Write a proxy mapper between inlangfs and vscode.workspace.fs for the ide-extension.

EDIT: I just read further up that you want to use a proxy that intercepts filesystem calls. I assume that you prefix/modify the path that should be resolved during the interception?

[openscript]

That's how it's supposed to be, but isn't the problem that $fs calls need to resolve from that path?

Yes. As a Inlang user I want to use relative paths inside my inlang.config.js. So the file system root needs to be, where the inlang.config.js is. If you have multiple inlang.config.js, because it's a monorepo, then we need multiple config instances and file systems.

EDIT: I just read further up that you want to use a proxy that intercepts filesystem calls. I assume that you prefix/modify the path that should be resolved during the interception?

No, it's not about the paths. We need a proxy mapper, because the vscode.Filesystem won't match the inlangfs. A few examples:

• vscode.Filesystem: readDirectory -> inlangfs: readdir
• vscode.Filesystem readFile has parameters (uri) which don't match with inlangfs readFile

[samuelstroschein]

Ah, and the filesystem implementation that we define should be

• async (because we are running on the main thread right now)
• defined functions resemble the API of node:fs/promises (?)

The methods

Proposal: (as you wrote) minimal surface area. if people need more APIs, they should open an issue.

• readFile
• writeFile
• readdir (used by some plugins, comes in very handy)
• more?

[openscript]

I agree, that it needs to be async and it needs the offered methods resemble node:fs.

It will be hard to offer more than the smallest fs offers:

So I would stick to readFile, writeFile and readdir (vscode FileSystem: readDirectory).

[samuelstroschein]

Agree. If people need more, they will open issues.

[samuelstroschein]

New filesystem API shipped recently.