RHEL 7 COMPILED VERSION ERROR: On Ubuntu 22.04 LTS and 24.04: "Invalid or missing encoding declaration" error message

[guss77]

After copying the content of the npbackup-gui archive from 3.0.0 RC to /usr/local/bin (not my preferred way of installing huge multi-file applications, but for the sake of following instructions to the letter), I run the command and immediately get an error window:

If I close it I get this text in the terminal:

$ /usr/local/bin/npbackup-gui 
Cannot create logfile. Trying to obtain temporary log file.
Message: [Errno 13] Permission denied: '/var/log/npbackup-gui.log'
Trying temporary log file in /tmp/ofunctions.logger_utils.log
[Errno 13] Permission denied: '/var/log/npbackup-gui.log'
Using [/tmp/ofunctions.logger_utils.log]
2024-06-11 00:50:54,386 :: WARNING :: Failed to use log file "/var/log/npbackup-gui.log", [Errno 13] Permission denied: '/var/log/npbackup-gui.log'
Using [/tmp/ofunctions.logger_utils.log].
2024-06-11 00:51:47,798 :: CRITICAL :: GUI Execution error invalid or missing encoding declaration for '/usr/local/bin/npbackup-gui'
2024-06-11 00:51:47,798 :: INFO :: Trace:
Traceback (most recent call last):
  File "/usr/local/bin/tokenize.py", line 385, in find_cookie
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 40: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/npbackup/gui/__main__.py", line 1032, in main_gui
  File "/usr/local/bin/npbackup/gui/__main__.py", line 871, in _main_gui
  File "/usr/local/bin/npbackup/gui/PySimpleGUI.py", line 10668, in __init__
  File "/usr/local/bin/npbackup/gui/PySimpleGUI.py", line 11363, in finalize
  File "/usr/local/bin/npbackup/gui/PySimpleGUI.py", line 11125, in read
  File "/usr/local/bin/npbackup/gui/PySimpleGUI.py", line 24730, in _refresh_debugger
  File "/usr/local/bin/inspect.py", line 1513, in stack
  File "/usr/local/bin/inspect.py", line 1490, in getouterframes
  File "/usr/local/bin/inspect.py", line 1464, in getframeinfo
  File "/usr/local/bin/inspect.py", line 782, in findsource
  File "/usr/local/bin/linecache.py", line 47, in getlines
  File "/usr/local/bin/linecache.py", line 136, in updatecache
  File "/usr/local/bin/tokenize.py", line 449, in open
  File "/usr/local/bin/tokenize.py", line 426, in detect_encoding
  File "/usr/local/bin/tokenize.py", line 390, in find_cookie
  File "<string>", line None
SyntaxError: invalid or missing encoding declaration for '/usr/local/bin/npbackup-gui'
2024-06-11 00:51:47,811 :: INFO :: ExecTime = 0:00:53.425603, finished, state is: critical.

[guss77]

Running under Plasma Wayland, here's the kinfo output:

Operating System: KDE neon Testing Edition
KDE Plasma Version: 6.0.90
KDE Frameworks Version: 6.4.0
Qt Version: 6.7.0
Kernel Version: 6.5.0-35-generic (64-bit)
Graphics Platform: Wayland
Processors: 20 × 12th Gen Intel® Core™ i7-12700H
Memory: 31.0 GiB of RAM
Graphics Processor: Mesa Intel® Graphics

[deajan]

Hello,

The tar archive should be decompressed in /usr/local/bin so you actually endup with /usr/local/bin/npbackup-gui/ directory containing the executable. But you can drop it where you want, there's no hardcoded directories.

The errors you're getting suggest permission issues. Your user isn't even allowed to write in /var/log

Can you try running as root to outrule permission issues ?

[guss77]

Ok, it wasn't clear to me from the instructions that I was supposed to copy the directory (I initially thought that I can copy just the npbackup-gui executable, but then I got errors about missing .so files so I just copied the content).

Extracting it into ~/.local as I preferred, and running it with specifying a user-accessible logging location:

$ ~/.local/npbackup-gui/npbackup-gui --log-file ~/.local/npbackup-gui/npbackup.log

I get this again:

And the terminal output:

2024-06-11 13:22:40,654 :: CRITICAL :: GUI Execution error invalid or missing encoding declaration for '/home/odeda/.local/npbackup-gui/npbackup-gui'
2024-06-11 13:22:40,654 :: INFO :: Trace:
Traceback (most recent call last):
  File "/home/odeda/.local/npbackup-gui/tokenize.py", line 385, in find_cookie
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 40: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/__main__.py", line 1032, in main_gui
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/__main__.py", line 871, in _main_gui
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/PySimpleGUI.py", line 10668, in __init__
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/PySimpleGUI.py", line 11363, in finalize
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/PySimpleGUI.py", line 11125, in read
  File "/home/odeda/.local/npbackup-gui/npbackup/gui/PySimpleGUI.py", line 24730, in _refresh_debugger
  File "/home/odeda/.local/npbackup-gui/inspect.py", line 1513, in stack
  File "/home/odeda/.local/npbackup-gui/inspect.py", line 1490, in getouterframes
  File "/home/odeda/.local/npbackup-gui/inspect.py", line 1464, in getframeinfo
  File "/home/odeda/.local/npbackup-gui/inspect.py", line 782, in findsource
  File "/home/odeda/.local/npbackup-gui/linecache.py", line 47, in getlines
  File "/home/odeda/.local/npbackup-gui/linecache.py", line 136, in updatecache
  File "/home/odeda/.local/npbackup-gui/tokenize.py", line 449, in open
  File "/home/odeda/.local/npbackup-gui/tokenize.py", line 426, in detect_encoding
  File "/home/odeda/.local/npbackup-gui/tokenize.py", line 390, in find_cookie
  File "<string>", line None
SyntaxError: invalid or missing encoding declaration for '/home/odeda/.local/npbackup-gui/npbackup-gui'
2024-06-11 13:22:40,662 :: INFO :: ExecTime = 0:02:40.275630, finished, state is: critical.

When running with sudo I get the exact same behavior as in the above details (i.e. no warning about the log file permissions, but otherwise the same failure).

[guss77]

BTW - the file /home/odeda/.local/npbackup-gui/tokenize.py does not exist - I'm guessing it is packed into the executable and the supposed path is an artifact of how it is extracted for execution, but it is pretty confusing.

[deajan]

Okay, running it in your home folder with --log-file resolves the permission issue, thanks.

TBH, I've no clue what could be the python problem here, I've ran 3.0.0RC1 on Debian/RHEL/Fedora systems so far.
Indeed, tokenize.py is baked into the executable.
Would you mind telling me what locale you're using ?
Also, could you run from console LC_ALL=C /path/to/npbackup-gui ?
Also, could you try to run npbackup-cli just to make sure it's not a console related issue ?

Did you run the GUI via a shortcut or via a shell ? If the latter is true, would you mind running it via another console app, or directly and a shortcut ?

The above will perhaps give me an insight.

I probably have to setup a KDE + wayland env to check if it appears that you're dealing with a GUI problem.
What is Operating System: KDE neon Testing Edition ? I thought KDE was a window manager, like Gnome. Do they make OSes now ?

[deajan]

BTW, updated install instructions as per your remarks.

A last test I could ask you is to install via python to a temp location you can delete later and test:

cd /tmp
python3 -m venv venv
venv/bin/python -m pip install npbackup==3.0.0rc1.post4`
venv/bin/npbackup-gui

This will allow me to outrule any compilation issues.

[guss77]

Would you mind telling me what locale you're using ?

I'm using the locale en_IL.UTF-8 - it is available on some Ubuntu based distros (like KDE Neon) when selecting Israel, but can also be generated by the standard tools. I do not believe it is an issue.

Also, could you run from console LC_ALL=C /path/to/npbackup-gui ?

Indeed it is the same output.

Also, could you try to run npbackup-cli just to make sure it's not a console related issue ?

The CLI does not run into the same problem. It seems to require an existing configuration (which I have yet to set up and I'm not yet sure what it means) but no obvious errors, except the requirement to manually set the log file otherwise it tries to write to the not user accessible /var/log and then complains about it (*)

$  ./npbackup-cli/npbackup-cli  --log-file ~/.local/npbackup-cli/npbackup.log
2024-06-11 14:57:31,268 :: INFO :: npbackup v3.0.0-rc1-4-pub-3.7-x64 2024060501 - Copyright (C) 2022-2024 NetInvent running as odeda
2024-06-11 14:57:31,268 :: CRITICAL :: Cannot run without configuration file.
2024-06-11 14:57:31,274 :: INFO :: ExecTime = 0:00:00.007345, finished, state is: critical.

I really feel that the default log file should in /tmp or some other use accessible location (~/.cache/npbackup.log maybe) - or do you expect npbackup tools to be run by the root user?

What is Operating System: KDE neon Testing Edition ? I thought KDE was a window manager, like Gnome. Do they make OSes now ?

KDE Neon is a build of latest (stable or testing) Plasma on the latest Ubuntu LTS and is provided by the KDE project as "best way to follow Plasma development". It is similar to GNOME's OS in scope, but unlike GNOME's that is meant only for testing (and I'm not sure how many GNOME devs are actually running it), Neon is intended for a wider audience and (at least on the stable branch) is considered daily-driver quality.

venv/bin/npbackup-gui

That seems to work fine.

I probably have to setup a KDE + wayland env to check if it appears that you're dealing with a GUI problem.

The GUI application runs under XWayland, so it probably doesn't need to know about Wayland, unless you'd want to use portals - which isn't currently the case.

[guss77]

Also:

Did you run the GUI via a shortcut or via a shell ? If the latter is true, would you mind running it via another console app, or directly and a shortcut ?

All the test above were done from the KDE terminal application Konsole. Using GNOME terminal or executing the command directly from a GUI has the exact same issue.

[deajan]

Thanks for your report.
I'm not expecting npbackup to run as root, it tries to write to /var/log, if it can't write there, it tries to get a temporary log file (/tmp/ofunctions.logger_utils.log which worked).
I'm not fond of writing log files by default into temp, since a sysadmin wants to know what happened without losing files.
Log rotation is also done so there's little chance to fill a disk.

The npbackup-cli indeed expects a configuration, generally done by npbackup-gui config UI. The CLI is designed to run on headless servers ^^

It's really interesting that using the non compiled (ie python npbackup-gui) version works.
It tends to confirm that I have a compiler issue with Nuitka (the compiler done by @kayhayen)

I've checked in the nuitka issues and found a similar issue, which has been closed for non feedback.
I'll check whether I can reproduce this error.

Thanks for your help.

[deajan]

It will take some time for me to get a reproducilble minimum example to submit to Nuitka's dev team.
In the meanwhile, could you test non compiled the python version of the program ?

[GuitarBilly]

[comment updated]

hello @Deejan,
FYI - I encounter exactly the same issue and syptoms on Ubuntu 22.04.2 LTS with Gnome and i3 window managers.
Previous version "v2.2.1-P7-x64 2023082801 - GUI enabled" runs on this machine without problems.

• npbackup-gui and npbackup-viewer give the invalid or missing encoding declaration.
• npbackup-cli works
• locale settings do not make a change. normally I run nl_NL.UTF-8 but I also switched to English US; same error.
• launching from gnome terminal or kitty gives the same error.
• when compiled via python then venv/bin/npbackup-gui works and shows the GUI without error.

Let me know if you need more information or trials.
I will use the working venv version to further test the application.

[deajan]

Got really bad internet right now, it took ages to get an Ubuntu spinned up.

But I was able to reproduce the issue.

Building directly in Ubuntu 22.04 LTS works:

I usually build on RHEL7 since the compiler (Nuitka) states that what is built on old systems works on recent ones, but not the contrary.
I have yet to find out why my RHEL7 build isn't liked by Ubuntu.

In the meantime, I've published an ubuntu compiled version at https://github.com/netinvent/npbackup/releases/download/v3.0.0-rc1/npbackup-linux-x64-gui-public-ubuntu.tar.gz

[guss77]

I'm not expecting npbackup to run as root, it tries to write to /var/log, if it can't write there, it tries to get a temporary log file (/tmp/ofunctions.logger_utils.log which worked).

That is legit, but there are two issues with that behavior:

1. If it fails to create a file in /var/log - it complains loudly and having that warning (apparently not an error) in the output is immediately pointed to as the problem when a user reports an issue (see comment 3 in this issue) 😅
2. It should fallback to a non-temporary $XDG_CACHE_HOME if available or $XDG_DATA_HOME if available before going to /tmp (see the spec here).

It's really interesting that using the non compiled (ie python npbackup-gui) version works.
It tends to confirm that I have a compiler issue with Nuitka (the compiler done by @kayhayen)

Various reports on the internet suggest that the problem is that a binary file is being opened as a text file, and since modern systems (like recent Ubuntu LTSs) default to UTF-8 as the system text codec, and unlike most 8-bit encoding UTF-8 has non-coding byte values (specifically, 0x80) python can fail to open some binary files in text mode. RHEL7's system text encoding is ANSI_X3.4-1968, which AFAIK maps all byte values to something, so it will open binary files in text mode without a problem.

I suggest forcing your RHEL7 build environment to UTF-8 mode by setting the LC_CTYPE environment variable to en_US.utf8, then finding the problematic open() and fixing it.

[katiuskt]

In case it helps:
Same issue observed in Fedora 40 (with GNOME 46)

File provided earlier (ubuntu compiled version)
https://github.com/netinvent/npbackup/releases/download/v3.0.0-rc1/npbackup-linux-x64-gui-public-ubuntu.tar.gz
does the trick

Worth to mention in Fedora 40 the warning message regarding the permissions in /var/log remains. Of course you can set the --log-file flag but as commented by @guss77 probably it would be a good idea to use a fallback directory.

[deajan]

Thank you for the input.
I'll sort the tmp log file issue out for RC2.
I must check whether I can succesfully build an universal binary with RHEL7 using LC vars.
I use to enter LC_ALL=C, but @guss77 remark is interesting.

But as said, I have very limited internet for the next weeks, so don't expect any (good) news until July ;)

[deajan]

So I've compiled with both ̀LC_ALL=CandLC_ALL=en_us.utf8` on RHEL7, but both builds fail on Ubuntu.
The only issue I've found so far is this at Nuitka.

I've checked all my open statements (corrected some missing encodings), but the exact error comes from PySimpleGUI.py library which is a 14kLOC hell.

i'll take any advice ;)

[deajan]

Here's what I've tried so far:

• Transform all py files from CRLF to LF (failed)
• Manually patch PySimpleGUI library (works, but is ugly)
• Upgrade Nuitka from 2.2.1 to 2.3.9 (has lots of unicode fixes, but doesn't work)
• Moving to FreeSimpleGUI (fails with v5.1.0 which is basically a blank fork, we have to compile Python 3.8 + tk for RHEL7 so we still get very old linux distros support)

I will at least create a PR for the patched PySimpleGUI here and probably here

[guss77]

I think the bug report in Nuitka is more or less relevant - as far as I can tell the problem is a weird interaction between Nuitka - that compiles Python code to a binary executable, and various things that want to read the source code that executes them for various reasons, and assume that is must be a textual source code (encoded in UTF-8, though there are some facilities to override that). This assumption of course breaks for Nuitka built code. This would also explain why the pip install works - it doesn't use a binary compiled executable.

Specifically the problem in PySimpleGUI is that it tries to setup some kind of debugger and uses inspect which in turn tries to parse its own executable. This will surely fail, but it will definitely fail to read the file because the tokenize.py module is specifically hard-coded to use the UTF-8 encoding to read content of "its source file" regardless of what LC variable is set to. Also because the problem is with reading compiler binary artifact - fixing things like new line mode and stuff won't help.

I think the best bet is to drop the debugging function somehow from PySimpleGUI - though I don't see a configuration option to control that, which I'm guess is what your patch is about: you didn't specify and I didn't find a PR about it. I tried some of the latest builds only only the "psgfix" build works for me.