Naming in tjs & subscoping

[KaruroChori]

As a follow-up of the discussion in #538.

Aside from this proposal, having a sub-namespace of tjs like tjs.posix. and covering (exclusively) all POSIX apis as close as possible would be great. However, for general usage, and for everything that is not POSIX, the interface could be simplified an regularized a lot.

API Entry	Suggestion (TBE is not yet evaluated)	Comment
addSignalListener	process.addListener
AF_INET	posix.xxx
AF_INET6	posix.xxx
AF_UNSPEC	posix.xxx
AI_ADDRCONFIG	posix.xxx
AI_ALL	posix.xxx
AI_CANONNAME	posix.xxx
AI_NUMERICHOST	posix.xxx
AI_NUMERICSERV	posix.xxx
AI_PASSIVE	posix.xxx
AI_V4MAPPED	posix.xxx
alert
args	process.args
availableParallelism	platform.threads	make getter
chdir	TBE
chmod	file.chmod
chown	file.chown
confirm
connect	TBE
copyfile	file.cp
COPYFILE_EXCL	file.cp.EXCL
COPYFILE_FICLONE	file.cp.FICLONE
COPYFILE_FICLONE_FORCE	file.cp.FICLONE_FORCE
cpuInfo	platform.cpu	make getter
createConsole	TBE	Its documentation is wrong. Not sure what it really does.
cwd	process.cwd
env	process.env
Error
errors	platform.errors
exec	process.exec
exepath	process.exepath
exit	process.exit
format
gc
getaddrinfo	TBE
gethostname	platofrm.info	make getter
getPid	process.pid	make getter
getPpid	-	Duplicated of ppid
homedir	platform.paths.HOME	We should have a specific module for well known locations.
inspect
IPPROTO_TCP	posix.xxx
IPPROTO_UDP	posix.xxx
isWorker	process.isWorker
kill	process.kill
lchown	file.lchown	Chown and lchown are quite similar. I would just have one + a flag
listen	TBE
loadavg	platform.loadAvg	As getter
lstat	file.lstat	Stat and lstat are quite similar. I would just have one + a flag
mkdir	dir.add
mkdtemp	dir.addTmp
mkstemp	file.addTmp
networkInterfaces	platform.networkInterfaces	make getter
open	file.open
pid	process.pid
platform	platform.type
PosixSocket	posix.socket
ppid	process.ppid
prompt
readdir	dir.read
readFile	file.read
realpath	paths.real	We should have a specific module for well known locations & utils like this.
removeSignalListener	process.rmListener
rename	file.rename
rm	file.rm
rmdir	dir.rm
SOCK_DGRAM	posix.xxx
SOCK_STREAM	posix.xxx
spawn	process.spawn
stat	file.stat
stderr	process.stderr
stdin	process.stdin
stdout	process.stdout
tmpdir	platform.paths.TMP	We should have a specific module for well known locations.
uname	platform.uname	make getter
unlink	file.unlink	does it make sense to expose it alongisde rm?
uptime	platform.uptime
userInfo	platform.user	make getter
version	-	I would avoid duplicating info from versions
versions
watch	dir.watch

[saghul]

Sorry it has taken me this long to provide feedback 😅

This is a tough one. Over time I have experimented with different approaches, buckle up.

Before I built txiki.js I built another engine using Duktape with a module system similar to Python: https://sjs.readthedocs.io/en/latest/modules.html

I liked it, but it gets annoying to have to import from like 5 namespaces for a simple script, specially since we are building a "small" runtime. ALso sometimes it's not 100% clear where some API needs to go and feels out of place everywhere.

Then, when I built txiki.js I wanted to try and do things simpler. Deno existed already, and I won't deny its influence. Initially I went with just 2 namespaces: core and stdlib, with core being what the runtime provides, and the stdlib are nice batteries included.

But also felt weird to use to I split it into what you see today: d13987e

Before that, I tried namespacing the stuff in the tjs namespace: 759dcad

If I think back, the reason why I likely went away from namespacing things is probably because there is always another way to organize them. Do we call it sys, process or platform ?

Last, both Deno and Bun have somewhat "proved" to me that a flat namespace has its benefits, and the arbitraty distribution of APIs in namespaces has no clear benefit to me.

[KaruroChori]

To me a flat namespace is a bit of a problem, because it does not group together symbols which are semantically linked, it makes harder to document features, and it is easier to end up later with collisions in naming which must be resolved with breaking changes.

Personally, I was considering sub-scoping tjs as

• fs
  • dir: open, list, chmod ...
  • file: open, quick read and write, chmod ...
• platform: versions, cpu, interfaces etc...
  • paths: (like home, tmp)
• posix: all basic functions mapping onto the posix standard & flags
• process: args, streams etc...

And to leave some symbols already there like gc (or to move it to engine, but it would be quite empty aside from versions maybe)
There is no need to have multiple imports, just something like this at the beginning:

const {fs, process} = tjs
//And you use fs and process without too much scoping

[saghul]

How are you using txiki.js? The problem I see here is that it's very subjective, and it's hard to have an objective point of view on all of the items above.

fs for example is pretty clear, but others are a bit more debatable. io could be one, or maybe net but which one to choose! 😅

The discoverability problem I don't see it as one, since all of things being in tjs means it's easy to discover them all.

Documentation... yeah that needs to be improved regardless of the chosen namespacing.

Regarding collissions, I don't think I've ran into that problem yet!

[KaruroChori]

How are you using txiki.js?

I will just answer this first, as it explains my motives :D.

In short, I use txiki as a runtime for applications where I need to glue together an extensive C/C++ codebase, and let people interface with it more easily. Doing it in node or bun is nowhere close as easy and friction-less. Also, I value a lot that quickjs is mostly reference counting when handling memory, and there is a lot of customization possible for the garbage collector, making it feasible for a class of real time tasks (as long as the linux kernel is patched).

At the moment I have two projects that I am working on and which are based on txiki:

• A series of services to run 3d printers & other robotic-adjacent applications. Some sort of klipper-like piece of software running on a host system controlling many MCUs, with the need to easily embed native code for some of the most computationally heavy numerical tasks.
• A custom distribution providing bindings to ggml, some of its derived projects and onnxruntime. So to have a powerful scripting environment to perform (mostly) inference work with a wide range of models and better covering low spec hardware compared to the overhead of other runtimes.

I considered, but cannot really use it as a pure scripting environment, because of the problems I have with tjs.x and the lack of native support for typescript. Otherwise it would be perfect for my linux embedded systems.

[saghul]

Thanks for the insight!

[KaruroChori]

Regarding collissions, I don't think I've ran into that problem yet!
The discoverability problem I don't see it as one, since all of things being in tjs means it's easy to discover them all.

Just because collisions have been hidden :D!
We have rm and rmdir. That would have been a collision but a suffix was assigned to one. In my opinion it would make much more sense to have file.remove/rm and dir.remove/rm
There are others which are just waiting for that to happen, like watch and open which as verbs are too generic to stand on their own. Or stat: what if I wanted to add usage stats of other resources? Or statistics related to the runtime itself?
That is why, I am so much of a proponent of namespacing:

• so that future changes or additions in the standard library are much more less likely to break the established interface
• and that by navigating through namespaces, the full chain of what does what on what is clear without needing any external documentation

The problem I see here is that it's very subjective, and it's hard to have an objective point of view on all of the items above.

Absolutely, the way we label and organize things is for the most part arbitrary and there is no clear metric to determine which one is best. All I can do is just to submit my proposal to be evaluated and hope for it to gain support :D.

This is the schema I modelled, with a short rationale for each top level subspace of tjs.

• fs any operation related to files, directories, and by extension devices because of how they are mapped on unix systems (but there is dev for proper "drivers").
• platform mostly readonly data explaining the platform on which the runtime is running. The OS, storage available, network interfaces, hostname, important paths, capabilities etc.
• engine any build specific information (compile flags), the list of external modules supported, embedded documentation & licences, stats, runtime configurable flags (gc for example)
• process any information about the process running. Its cwd, args, the environment variables set, information about its affinity etc.
• session any information which is persistent and should survive across processes when the same application is run. Basically something related to window.localStorage
• posix posix interfaces which are left without a better JS mapping, and constants which cannot be scoped because of that. Basically the graveyard of necessary ugliness.
• dev would be there for further sub-scoping of custom builds and modules which are adding device driver functionality. For example, my extension to use serial ports would be scoped in tjs.dev.serial. tjs.dev.gpio for GPIO. The LVGL virtual display would add functions to tjs.dev.lvgl

I am not really saying it is the one and right way to do it, but I think it is a sound approach.
But again, I am not really trying to force it onto txiki. At least not on this repo :D.

[saghul]

Moving from the other dicsussion:

Still, I am more than open to discuss them early to keep the amount of divergence to a minimum.
As for the interface, I think there are some objective flaws. They all have limited severity, but the compounded effect makes the experience a bit frustrating:

Redundancy of some elements

What do you mean by this one?

Sometimes objects are exposed, sometimes functions returning objects.

What would those be? Making things consistent there would make sense indeed.

Inconsistent casing

Yes, this can be fixed!

Limited documentation of most entries

Yeah, I agree, things should be better documented 😅

Lack of versioning guidelines to ensure new releases are compatible with older code (which goes back to the issue I have with versioning)

What's the issue with versioning? Right now there is zero guarantee about backwards compatibility. As you might have noticed, not everything is cast in stone. If you are only using web APIs you should be fine, but the tjs namespace is "unstable" with regards to its API.

I feel like this project is small enough to not need strong versioning and API guarantees yet, since I am still figuring stuff out all the time, there is no master plan here :-)

If / when the core APIs have been stable for long enough that it makes sense to make some guarantees / promises, I'd be happy to do that!

[KaruroChori]

What do you mean by this one?

I commented some in the first message. For example version/versions, ppid/getPpid or more gray cases like lstat&stat which should be reduced to one function and a flag in my opinion.

What would those be? Making things consistent there would make sense indeed.

I also tagged many of them in the table. Like availableParallelism, gethostname ... the list is quite long.

What's the issue with versioning? Right now there is zero guarantee about backwards compatibility.

Exactly, that is the main issue for which I was planning to diverge :D.
It is not about fixing things into stone, but to make people downstream aware that there has been a breaking change. Semantic versioning does that for free based on an implicit contract derived from the versioning scheme, and a lot of the modern tooling will take that into account automatically.
Not doing that risks to hinder the development of any ecosystem from my point of view.

[saghul]

I commented some in the first message. For example version/versions, ppid/getPpid or more gray cases like lstat&stat which should be reduced to one function and a flag in my opinion.

• version/versions: there is very little reason to check versions, it's more like build information, so perhaps it can be moved to some build object with more related things.
• ppid / getPpid: that's an oversight! When I added the getter I forgot to "blacklist" the function so it's not exposed in the core
• stat / lstat: both Deno and Node use separate APIs, so I'd say it's reasonable to keep them separate

I also tagged many of them in the table. Like availableParallelism, gethostname ... the list is quite long.

Got it, even though it may not seem like it, there was a system lol: posix APIs got their posix name, the others I JS-ified it. But I agree, let's make it right!

Exactly, that is the main issue for which I was planning to diverge :D.
It is not about fixing things into stone, but to make people downstream aware that there has been a breaking change. Semantic versioning does that for free based on an implicit contract derived from the versioning scheme, and a lot of the modern tooling will take that into account automatically.
Not doing that risks to hinder the development of any ecosystem from my point of view.

I can agree to that, yeah, but as you can see from this very interaction the project is too young for that IMHO. Semver wise we'd bet at 0.X.Y.

[saghul]

Started some small steps in #583

[KaruroChori]

Great!
If you want I can prepare a checklist, just with those changes we both agree upon or which are likely supported by both.
So we can quickly review all of them and split the refactoring chore.

[saghul]

Sounds good! I came up with new ideas while cycling this morning, I'll try to get to them tonight!