feat: i18n framework based on extracting translations from plugin's /languages/ to /language/

[linonetwo]

Sorry for being late.

This is based on logic in #7821 (comment)

Lingo macro keep untouched. Will handle fallback between de-AT and de-DE. fixes #8263

How it works:

1. When wiki start and language change, $tw.wiki.unpackPluginTiddlers will be called, and in this method we can modify shadowTiddlers, which will be used by $tw.wiki.getTiddler from wikitext.
2. I add a call to my new $tw.utils.activatePluginTranslations method in unpackPluginTiddlers
3. Plugin authors will write l10n tiddlers under xxx/languages/, and this PR make TW auto move it to xxx/language/, so lingo macro works with xxx/language/ base path.
4. It move en-GB first, then the active language and dependeicies, so it will fallback in cascade.

1. RFC: $tw.utils.activatePluginTranslations will be undefined on first call, when utils is not mounted, so I check if it is undefined
2. RFC: when loading wiki and switch language, unpackPluginTiddlers will be called at least 3 times, wasting lots of repeated work in activatePluginTranslations. But I can't skip it, because it reset shadowTiddlers = Object.create(null); each time it's called.

Downside of this implementation is it not able to translate user wiki any more. Which was achieved by modify lingo macro in #7821:

!! Any l10n strings

To translate any text that directly placed in user's wiki, instead of in a language plugin, you can set the lingo-base variable to teach <<.def lingo>> macro the place to look for.

[github-actions]

Confirmed: linonetwo has already signed the Contributor License Agreement (see contributing.md)

[pmario]

I did just review the code. I'll give some more feedback. Once I did test it in detail.

[linonetwo]

I also tried patch one plugin from another, with fallback.

[pmario]

@linonetwo -- I thought you would have a closer look at the $:/core/modules/pluginswitcher.js, which is responsible to switch languages and themes.

I intended to suggest that the new functionality of extracting and removing languages should be part of the pluginswitcher.js

It is registered at startup.js but the main code is at /pluginswitcher.js

• At startup.js line 80 the language-switcher is registered, which in turn adds an event listener from pluginswitcher.js
• The swtichPlugins() funciton contains the logic to extract the language tiddlers from the corresponding plugins and overwrites the default en-GB shadow tiddlers, because they have the same "namespace"
• accumulatePlugin() takes care of the dependencies
• Then there is .unregisterPluginTiddlers()
• . registerPluginTiddlers()
• and finally .upackPluginTiddlers()

This mechanism makes sure that only those shadow tiddlers are in the shadow-store that are part of the activated language.
If a language is missing some translations, the default language or dependents are still there. Because of the same namespace.

IMO there is no reason to have all languages extracted from the plugins at startup. The number of language related shadow tiddlers should be constant and as low as possible.

We only have 1 core plugin in this PR, but the number of shadow tiddlers is increased by 174 tiddlers. The menubar-plugin de-DE translations.multids only contains 25 new texts + readme it should be 26 shadows more and not 174.

We have 33 possible languages in TW, which makes a total of 858 potential shadow tiddlers for menu-bar plugin alone.

The current total number of TW shadow tiddlers is: 2476 for v5.3.5 -- With the implementation in this PR the menubar has the potential to increase the number of shadows by 35%, which will directly impact the core performance for every filter that uses [all[shadows+tiddlers]] -- That's basically every UI related tiddler.

I was not happy, that every possible language is part of the plugin (because of size). -- But OK -- It was decided, that that's not a problem.

But I definitely think that only 1 set of language shadow tiddlers should be active at any given time.

[linonetwo]

every possible language is part of the plugin

In current implementation, you can use another language plugin to provide l10n. See my last comment.

• The swtichPlugins() funciton contains the logic to extract the language tiddlers

AFAIK It does this by calling unpackPluginTiddlers, which is I'm currently using. And in swtichPlugins() there is no way to access to shadowTiddlers. So I can't put my logic in it.

number of shadows will directly impact the core performance

In unpackPluginTiddlers -> activatePluginTranslations, I can delete unused shadowtiddlers.

[pmario]

number of shadows will directly impact the core performance

In unpackPluginTiddlers -> activatePluginTranslations, I can delete unused shadowtiddlers.

I think it should not be needed to delete unused shadow tiddlers. Only used shadows should be extracted into the /language/ namespace. As long as no unnecessary tiddlers are extracted no shadows need to be deleted.

en-GB will need to be always extracted since it is the "source of truth" and the final fallback. If the $:/language setting is de-DE then the German tiddlers are extracted into the same namespace that en-GB already uses. So no extra shadows are ever created. The same should be true for every other language, that is part of a plugin.

[linonetwo]

@pmario I've update the logic in unpackPluginTiddlers to not unpack unused language tiddlers.

The logic have to be called in unpackPluginTiddlers instead of in PluginSwitcher, because other places that call the unpackPluginTiddlers will clean up the language shadow tiddlers, so language shadow tiddlers added on language plugin switching/startup will lost.

I'm using $tw.language.activatePluginTranslations because I don't want to make boot.js file become much longer. Or we can use a library module and require it here, instead of use a util and check its existance.

[pmario]

@linonetwo

I do like your translation concept better than my initial one, which is much more complex. 🥇

I think the plugin menubar/language/ directory should be renamed to /languages/ to be consistent with the tiddler namespace.

The code and functionality looks good to me now.

every possible language is part of the plugin

In current implementation, you can use another language plugin to provide l10n. See my last comment.

I did miss that. That's a good thing. So we could decide to change the current behaviour if we want to, without compatibility problems.

@Jermolene

I did test:

• The dependents setting -> OK
  • The good thing is, that dependents only need to be defined once in the main language file. Plugins will inherit this configuration.
• Switching languages to German de-DE <-> de-AT -> OK
• de-AT only contains 2 translations -> uses fallback de-DE -> OK
• Change language -> Restart the wiki -> OK
• Change back to English -> OK
• Set language to de-DE and remove Config/HeadingMainXXX so en-GB fallback needs to be active -> OK

[pmario]

@linonetwo -- Please keep my de-AT and de-DE translations within this PR. So it's easier to test. -- I'll close my menubar-translation PR.

[linonetwo]

I think the plugin menubar/language/ directory should be renamed to /languages/ to be consistent with the tiddler namespace.

I was using /language/ because I wasn't delete unused shadow tiddlers. Now that they are deleted, it is OK to put then all under /languages/.

But notice that core translations are like {{$:/language/Buttons/Underline/Hint}}, and they are loaded from $:/languages/en-GB with /languages/. So it is also reasonable to keep the both /languages/ and /language/.

[pmario]

I was talking about the physical path name on the harddisk. The JS code is OK

[linonetwo]

@Jermolene @pmario Is current implementation good enough?

[pmario]

See my last post: #8435 (comment)

[linonetwo]

@pmario isn't that updated in last commit?

[Leilei332]

Would it be possible to seperate plugin language packs from plugin itself? IMO this can avoid unnecessary language to be installed and avoid the size of the tiddlywiki file be increased too much when the plugin has many languages.

[linonetwo]

Would it be possible to seperate plugin language packs from plugin itself?

@Leilei332 Yes. Mario already ask same question above.

[linonetwo]

@Jermolene Need decision on this, more and more my plugins are providing i18n, so need to decide a standard way for them before they are getting too much.

[Jermolene]

Hi @linonetwo this PR is very important and I apologise for not giving it attention until now. I do think the approach is broadly correct. There's a lot of brittleness from the dependency on the plugin naming convention which I think we need to address (see below)

[Jermolene]

The dependency on the format of plugin names is very brittle.

An option we should consider is to extend the plugin format to include languages in additional properties to the existing "tiddlers" property:

{
	"tiddlers": {
		"$:/config/DefaultColourMappings/menubar-foreground": {
			"title": "$:/config/DefaultColourMappings/menubar-foreground",
			"text": "#fff"
		},
		"$:/config/DefaultColourMappings/menubar-background": {
....
	"languages": {
		"en-GB": {
			"$:/plugins/tiddlywiki/menubar/readme": {
				"title": "$:/plugins/tiddlywiki/menubar/readme",
				"text": "This is English"
			}
		},
		"fr-FR": {
			"$:/plugins/tiddlywiki/menubar/readme": {
				"title": "$:/plugins/tiddlywiki/menubar/readme",
				"text": "This is French"
			}
		}
	}

On the Node.js side there would need to be some reorganisation so that the language tiddlers can be distinguished from the ordinary plugin tiddlers. One approach might be to allow plugins to opt-in to a new folder format where there is are "tiddlers" and "languages" folders within the plugin folder. Perhaps

[linonetwo]

I was reading science fiction novel A Deepness in the Sky last night, didn't saw this. I will put more time here in next few days.

One approach might be to allow plugins to opt-in to a new folder format where there is are "tiddlers" and "languages" folders within the plugin folder. Perhaps

You mean when dev plugins? But TiddlyWiki way is very flexible, use metadata to describe the tiddler, don't care the physical location of it. Is it worthwile to break this?

Adding a new "languages": field in JSON plugin will mark them, but how do we know it belongs to there when packaging plugin JSON? By a field or file type? Then we can simply use this information in boot.js:1496.

I suggest add a new tiddler type text/vnd.tiddlywiki-language or text/vnd.tiddlywiki-lingo or text/vnd.tiddlywiki-localization or text/vnd.tiddlywiki-l10n, or lingo/vnd.tiddlywiki (This is valid as GPT said...)

title: $:/anything/xxxx/yyy/
type: text/vnd.tiddlywiki-localization

CloseAll/Button: close all
ColourPicker/Recent: Recent:

Don't need to add /language or /languages in the title, while this is still recommended so it looks organized in "tree" tab of plugin.

[linonetwo]

Oh, no, they can't have same title, otherwise they will overwrite each other in constituentTiddlers.

Difference from language plugins, plugins are enabling every language tiddlers at the same time.

Then how about we keep the convension on $:/languages/, have convension here will make the code and format much simpler.

title: $:/languages/en-GB/plugins/linonetwo/xxx/

CloseAll/Button: close all
ColourPicker/Recent: Recent:

so we can check them by prefix $:/languages/, instead of constituentTitle.split('/')[4] === 'languages'

[linonetwo]

Or we can remove the "skip" here, it only reduce the count of shadow tiddlers. It doesn't matter if we still load them. Or use type: text/vnd.tiddlywiki-localization to skip them, but we keep the old logic in core/modules/language.js , do you like the logic modified in core/modules/language.js ?

(But have to admit that I'm afriad of modifying plugin packing and unpacking so I said this.)

[linonetwo]

I'm developing a game that learn the tiddler concept of TW, in that game each plugin can declare patch-plugin: ["some other plugin"]. And I process the patch dependency tree before load them. So I know a plugin is being patched by A and B, I will process them together.

If it also works here, I can know language plugin $:/languages/de-DE is being patch by community plugin $:/A/B, so when loading $:/languages/de-DE, it can also load a tiddler from $:/A/B that says

title: $:/language/
patch-plugin: $:/languages/de-DE

Blabla: Blabla

And push these tiddlers right into constituentTiddlers of $:/languages/de-DE plugin,(also record the patching chain in a field patch: $:/languages/de-DE $:/A/B, so plugin sourcing is easier.), and just load and use $:/languages/de-DE as if it is untouched.

Just sharing my design in my system here, not intended to introduce it to TW.

[linonetwo]

@Jermolene I think we also need to consider i18n for doc site. (Here is an exmple) Current implementation make it impossible to create multiple version of documentation, I just aware of it, and can only switch plugin's language.

So the way that adding new structure only on plugin will also weaken the i18n framework. While the old runtime way in #7821 works with doc site.

Can we use a new field or new format (like TestCaseTiddlers) or new file extension (like .multid) to mark a tiddler as l10n file. During boot and switching language, we do the extraction based on this info.

BTW, can we edit text/vnd.tiddlywiki-multiple inside wiki? Or they are like .multid file that can only be edit before boot. This depends if we can edit l10n file inside wiki.

[pmario]

@Jermolene I think we also need to consider i18n for doc site. (Here is an exmple) Current implementation make it impossible to create multiple version of documentation, I just aware of it, and can only switch plugin's language.

Why make it complicated, if it can be easy. -- Just create 2 editions and link them. A CI/CD process can handle the automatic creation -- done.