Non-latin characters are displayed as '?'

[Levitanus]

Linux 5.11.0-49-lowlatency #55-Ubuntu SMP PREEMPT Wed Jan 12 18:18:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Reaper 6.44, sws-2.12.1.3-Linux-x86_64

I see some troubles of unicode rendering. At the first glance I thought this is a problem of the selected font, but tests with Cyrillic font (as from python as from lua) showed it is not a matter.

Later I thought it is connected to the way strings has to be packed for the C-interface. In the reapy project, we've made hacks of original reaper_python for correct working of various strings and pointers. But, as far as I tried (not very long) — I didn't succeed with the problem.

As long as I can see it is not a issue of reascript itself, as Russian characters in the track name are visible as from ReaConsole as from the print statements (in the track manager example), but not in ImGui window.

So, at the moment, I think the problem is on the ReaImGui side. Later I'll try to make a minimal working unicode text example with JS_* API to test whether some characters can be drawn or not.

I'll carefully say Reaper and SWELL leaks some points of unicode support, generally, in Chinese symbols. But Russian works well.

---

The code below uses wrap I'm preparing, but It does exactly the same as original python port.

import reapy as reaper

ctx = reaper.ImGui.CreateContext('My script')
size = 25
font = reaper.ImGui.CreateFont('/home/levitanus/Загрузки/fonts/Montserrat/Montserrat-Light.ttf', size)
reaper.ImGui.AttachFont(ctx, font)

click_count, text = 0, '(例子) - chineeze, (рыба) russian'


def frame():
    global click_count, text

    if reaper.ImGui.Button(ctx, 'Click me!'):
        click_count = click_count + 1
    if click_count % 2 == 1:
        reaper.ImGui.SameLine(ctx)
        reaper.ImGui.Text(ctx, 'hello dear imgui!')

    rv, text = reaper.ImGui.InputText(ctx, 'text input', text, 1024 * 2)


def loop():
    reaper.ImGui.PushFont(ctx, font)
    reaper.ImGui.SetNextWindowSize(
        ctx, 400, 80, reaper.ImGui.Cond_FirstUseEver()
    )
    visible, open = reaper.ImGui.Begin(ctx, 'My script', True)
    if visible:
        frame()
        reaper.ImGui.End(ctx)

    reaper.ImGui.PopFont(ctx)

    if open:
        reaper.defer(loop)
    else:
        reaper.ImGui.DestroyContext(ctx)


reaper.defer(loop)

[cfillion]

At this time, ReaImGui rasterizes glyphs only from the Basic Latin and Latin Supplement Unicode blocks (U+0020 to U+00FF). UTF-8 is fully supported internally, however characters outside those blocks are displayed as '?'.

Dynamically finding and rasterizing glyphs on an on-demand basis is not currently possible, until a solution like ocornut/imgui#3471 gets finished and merged into Dear ImGui.

A temporary workaround could be to allow scripts to customize the range of characters to load from the font. However that still won't let them render all possible character from the entire Unicode, as loading everything would be slow and take way too much video memory.