feat: t macro and docs

[linonetwo]

See #8435

---

Docs in Preview site

The old lingo macro can only work with language plugin, which is very restricted, we plugin developers don't have a chance to use it easily.
The new t macro make it flexible for plugin developers (or even normal users that want their blog to be international) to do translation.

The implementation is simple, works in user-land, and is compatible with infra like i18n ally. (Possibly can use with i18n ally with inline annotation, if lokalise/i18n-ally#1021 is solved.)

fixes #4179 fixes #786 closes #5418

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jun 8, 2024 11:48am

[linonetwo]

At some related discussion members @pmario @Gk0Wk @sycom @joshuafontany @tobibeer

[sycom]

Pretty interesting. Some questions

• if you don't use a translation plugin, how does the wiki know which language should be displayed ?
• can you confirm that with this macro anyone create his own translation ? Which is great.
• but I guess that the original tiddler should be prepared to permit it ?

Not exactly the same purpose, but I've been working on translation lately. Just have not enough time to publish and comment, sorry for that. I'll do a push asap on my repo. Will tell you.

My approach is a bit different: it's a plugin that enables translation capabilities. Translations are fully independent. You can translate any tiddler without modifying original one.

[linonetwo]

• how does the wiki know which language should be displayed

<$set name="lang" filter="[[$:/language]get[text]get[name]else[en-GB]]" select="0">

• the original tiddler should be prepared to permit it ?

What does this mean?

This macro is just a simple string replacement like https://www.i18next.com/ which is widely used in modern frontend dev in my daily work.

[pmario]

Do I need to install all possible translations, if I do install the plugin? -- If yes that's an absolute "no go" for me.

[sycom]

<$set name="lang" filter="[[$:/language]get[text]get[name]else[en-GB]]" select="0">

Ok I get it. You have to install the language plugin to switch from one language to another.

What does this mean?

Sorry for being unclear. My English is a bit odd... I understand that only predefined tiddler or part of tiddlers can be translated. So will you be able to translate someone else's plugin that didn't anticipate the translation ?

[linonetwo]

Do I need to install all possible translations, if I do install the plugin? -- If yes that's an absolute "no go" for me.

No, as you can see, texts are in normal tiddlers, you can pack them to a separate plugin. Just make them sub-plugin.

[linonetwo]

You have to install the language plugin

language plugin is for tw cores translation. here we just read the language config.

you don't need to install additional plugin to use translation of your plugin, if you put all translations in your plugin.

[sycom]

Thanks for your answers. So if I get all this correctly, the macro offers possibilities without killing existing ones which is good. I don't see problematic behaviour. I'm not "core fluent" enough to evaluate if it's worth adding it to the core though.

If your macro is not integrated to the core, you may publish it elsewhere. It will surely be interesting for other users.

For your information, as mentioned above, I pushed my first release of t8n plugin if you want to check this alternative approach. Code on codeberg : https://codeberg.org/sycom/TiddlyWiki-Plugins

[Jermolene]

Thanks @linonetwo, and sorry to have missed this when you first posted it. I like the general idea here, but have a couple of thoughts:

• We should use procedures for this, rather than macros. There is also a good case for making the functionality available via a function as well
• The core should not use generic names like "t", I think of single character variable names as being reserved as abbreviations that can be used by end users. Perhaps "translate" or conceivably "xlingo"
• TiddlyWiki doesn't use the abbreviation "i8n" but instead the word "lingo"

[linonetwo]

It takes me 10 min to learn and refactor to \procedure, and 1 hour to learn & refactor to \function . It was a good introduction exercise for me...

I also update the doc, see it in "Docs in Preview site" at #7821 (comment)

@Jermolene Please check this again.

[linonetwo]

@Jermolene Hi, please take time for this, every time I develop a plugin, it reminds me of this.

[Jermolene]

Hi @linonetwo thank you for the updates.

Thinking again, we do not need to introduce a new procedure here. I think we can overload the existing <<lingo>> macro like this:

<!-- Note that lingo-base should end with a trailing slash character -->
\procedure lingo-base()
$:/language/
\end lingo-base

\procedure lingo(title,override-lingo-base)
<!-- Get the parse mode used to invoke this procedure -->
<$parameters $parseMode="parseMode">
	<!-- Compute the lingo-base-->
	<$let active-lingo-base={{{ [<override-lingo-base>!match[]else<lingo-base>] }}}>
		<!-- First try the old school <active-lingo-base><title> format -->
		<$transclude $tiddler={{{ [<active-lingo-base>addsuffix<title>] }}} $mode=<<parseMode>>>
			<!-- If that didn't work, try the new <lingo-base><langcode>/<title> format -->
			<$let language-code={{{  [[$:/language]get[text]get[name]else[en-GB]] }}}>
			<$text text={{{ [<active-lingo-base>addsuffix<language-code>addsuffix[/]addsuffix<title>] }}}/>
				<$transclude $tiddler={{{ [<active-lingo-base>addsuffix<language-code>addsuffix[/]addsuffix<title>] }}} $mode=<<parseMode>>/>
			</$let>
		</$transclude>
	</$let>
</$parameters>
\end lingo

A couple of changes from your version:

• The lingo-base should have a trailing slash, for consistency with existing usage
• Rather than a mode parameter, we pick up the parsing mode that was used to invoke the transclusion

One benefit of this approach is that it allows us to use <<lingo>> in the caption field of core tiddlers, thanks to your fallback parameter idea.

I also think that it would be good if we could include add translations for one of the core plugins to act as an exemplar of best practice. I would suggest the "tiddlywebadaptor" plugin, but would be open to suggestions. Perhaps you could include the Chinese translations, and then others can contribute other languages.

[linonetwo]

Very interesting approach, I've updated the definition and the doc. These features are even advanced for me, it is the first time for me to use them.

@Jermolene new doc is in https://tiddlywiki5-25onfr134-jermolene.vercel.app/#lingo%20Macro , let's see if it is clear.

[Jermolene]

Hi @linonetwo I think this is ready to merge. Would you please be able to resolve the merge conflicts?

[linonetwo]

Cool, will do it now, I was playing game during duanwu festival last few days.

[linonetwo]

I added

title: $:/plugins/tiddlywiki/menubar/tree
type: text/vnd.tiddlywiki

<<tree prefix:"$:/plugins/tiddlywiki/menubar/">>

to demonstrate the tree structure of i18n file:

But consider the potential change will made in #8234, I decided not to include it in this PR.

---

I want to add it back, because in the doc I add a link to help user understand the structure:

---

Now hope this is ready for merge.

[Jermolene]

Thanks @linonetwo

[Jermolene]

Hi @linonetwo I think the problems revealed in #8262 and #8263 will require significant rework to get things working. It is entirely my fault for suggesting this approach in the first place. I did not consider how the language plugin switching mechanism interacts with language dependents mechanism.

Although ingenious, the fix in #8269 is not sufficient because it does not work where there are more than one dependent language plugin, or if the dependencies are nested more than two levels. I suspect that an effective fix will likely require JS changes to the language plugin switching code.

I am very sorry for the disappointment but I propose to revert this PR for v5.3.4 so that we can work on a robust implementation for a subsequent release.

[linonetwo]

So maybe use JS to rewrite lingo macro? Writing JS version will be very easy for me, We can keep the same marco params, but use JS to cache translation from plugin to speed up rendering, also handle fallback cases.

more than one dependent language plugin

I think this case is very rare, why not handle this in next release, but provide this feature in this release, as long as the API won't change. So plugin authors can start writing i18n. I can also write a JS macro version, to see if we can let this feature catch this release.

But it's indeed good to delay this to next release, so it's not so hurry. I only PR for fun, this feature exists as patch in TidGi, so this doesn't affect my daily usage.

[Jermolene]

Hi @linonetwo I understand the arguments but my concern is that the eventual fix for this may not be backwards compatible with this partial fix.

You will be familiar with the way that much of the logic for handling language plugins is in this JS function:

https://github.com/Jermolene/TiddlyWiki5/blob/8eb08820ac468aff17caf04cff35a6920a104e9d/core/modules/pluginswitcher.js#L40-L75

One reasonable approach to consider would be for the language plugin switching code to somehow also harvest translation strings from other plugins if they are present. With such a solution, the language strings would need to be marked in some way, and perhaps require other fields. None of that would be compatible with the partial fix here, but it would be quite efficient and congruent with the existing architecture.

So, I think this needs more thought and experimentation before we can propose a solution.

[linonetwo]

@Jermolene I will write a JS macro that consider the usage in pluginswitcher.js soon, just need you to decide:

1. will it be a JS macro?
2. reuse wikitext lingo macro's interface?
3. should it cache the i18n result from system tiddlers (they usually don't change during wiki usage, only change when user upgrade a wikitext plugin)

[Jermolene]

@Jermolene I will write a JS macro that consider the usage in pluginswitcher.js soon, just need you to decide:

Hi @linonetwo I am not sure that that is necessarily the best way to go. I wonder if there is an alternative approach where we leave the existing macro untouched, and do the work of folding in add-on translations upfront, in the switchPlugins method. It would work like the "$:/info/*" mechanism, which generates a temporary plugin $:/temp/info-plugin. We'd collect up all the add-on translation strings that match the current language name, or any of the dependents, and pack them into the plugin under their usual title.

[linonetwo]

@Jermolene I'm learning Lean4 language these days, but I might write a demo at weekend.

Where will the translation go? In previous design, they will be placed at $:/plugins/author/name/languages/zh-Hans/xxx. Do you mean they should also have a tag $:/tags/Language (or we can use $:/plugins/author/name/languages/ as a convension path)

And these tiddlers are being clone to $:/language/author/name/xxx (only in memory at runtime, no save-back to fs, so maybe it has to be $:/temp/plugins/author/name/languages/zh-Hans/xxx) when switching language or on booting?

@pmario You wrote the #5418 , what's your opinion here?

[pmario]

@pmario You wrote the #5418 , what's your opinion here?

---

My initial concept was completely different.

I think your code from https://github.com/linonetwo/TiddlyWiki5/blob/e9568720489ae5fe9ee74cd16330b8168028beff/core/wiki/macros/lingo.tid could be changed in a way, that it can resolve several dependents with several levels deep.

But imo it would need a recursive wikitext procedure, with an "endless loop" protections, which makes it much more complex and hard to maintain.

TLDR; I think the mechanism can be even simpler than the one Jeremy suggests at: #7821 (comment)

---

I think it will be possible to keep the directory structure and the plugin naming conventions in place. But instead of "extracting" every plugin tiddler at startup and have a complex lingo-macro it should be simpler.

At wiki startup and if language is switched

1. We extract plugin en-GB language tiddlers to the $:/plugins/tiddlywiki/menubar/language/ "namespace"
   1.1. That's the automatic fallback. No extra fallback handling needed
2. Then we check the $:/language tiddler
3. If it is zh-Hans we extract everything we can find under: $:/plugins/tiddlywiki/menubar/language/zh-Hans/ -- and
4. Extract it to: $:/plugins/tiddlywiki/menubar/language/
   4.1. So the en-GB shadows are overwritten if the translations exist
5. Then we check the core-language-plugin settings for "dependents" and we resolve them one by one.
   5.1. The order is important - Tiddlers need to be extracted in reverse. eg: if language is de-AT which depends on de-DE -> de-DE has to be extracted first. -> So de-AT wins
   5.2. go to 4. until we are done
6. So now the lingo-macro can be as simple as it is now

The good thing will be, that the ControlPanel -> Plugins -> MenuBar -> Contents tab will only show
$:/plugins/tiddlywiki/menubar/language/* tiddlers, that are in use.
This will be much more convenient to support once plugins will be translated to 30+ languages.

The $tw.languageSwitcher and the $tw.themeManager core code are both instances of $tw.PluginSwitcher.

• See: $:/core/modules/pluginswitcher.js https://github.com/pmario/TiddlyWiki5/blob/master/core/modules/pluginswitcher.js
• https://github.com/pmario/TiddlyWiki5/blob/master/core/modules/startup/startup.js#L78-L107

The .themeManager is already able to add and remove shadow tiddlers, if needed. The .languageSwitcher should be able to do the same thing.

[linonetwo]

• we extract everything we can find under: $:/plugins/tiddlywiki/menubar/language/zh-Hans/ -- and

• Extract it to: $:/plugins/tiddlywiki/menubar/language/

Original i18n tiddlers should be placed under $:/plugins/tiddlywiki/menubar/languages/zh-Hans/, note the /languages instead of /language, otherwise this namespace will not only contains zh-Hans en-GB but also extracted keys lingoKeyaaa lingokeybbb ...

I think the approach is clear now, this also sometimes allow directly transclude the tiddler, instead of need to use lingo macro, which sometimes need a "wikify".

I will take time for this, but I'm going to build an AI character chat plugin this weekend because I just gain lots of interest on it.

[linonetwo]

#8435