Improve packaging procedure

[wvengen]

Artisan is a complex piece of software with many dependencies. To make it work easily for users across various platforms, bundled installations are provided. Recently it was also released as a Flatpak (#1602), which required a somewhat different, out-of-tree, installation procedure.

Also noticing that packaging scripts for the different platforms have some overlap, I'm working out to see how the packaging could be improved. My goal is to:

• better support non-bundled installations
• cleanup the Flatpak build steps (moving build steps from its manifest to the Artisan source tree)
• (hopefully) also improve the packaging scripts (incl. bundling), decrease duplication, make it easier maintainable
• (as a bonus, maybe) reduce the size of bundled installations (re-using the lessons learned from creating the Flatpak)

Note that this issue will probably take quite some time, as bundling often has many thorny sides to it, and it needs testing on all platforms. But I'd like to try to see if this can be improved.

[wvengen]

For a start, I've looked at what some other PyQt-based projects are doing, and put it in a spreadsheet: pyqt-packaging.ods

Projects: Vorta, Ginga, Frescobaldi, ReText Markdown editor, ToirtoiseHg, ViTables, OnionShare, Anki client, RazerCfg, QWebOrf, QR Tools. (Note that I have a bit of a Linux bias, please suggest other projects that we could learn from.)

Some main findings:

• almost all can relatively easily install from source, many also directly from PyPi
• Linux packages are usually not bundled, Mac OS and Windows are
• Linux desktop and Flatpak installation often work with the package/wheel + desktop files
• bundling scripts are often custom
• i18n and qrc building is always custom, sometimes part of setup.py
• applications that also provide a library can usually choose between PyQt5, PyQt6, PySide2, PySide6

[wvengen]

Seeing that each has their own way to work with i18n and qrc building, it may be a good idea to make this a Python package.
And there is: setup_qt, pyqt-distutils and distutils-ui, though they're really old and don't support PyQt6.

[MAKOMO]

We are about to release a new version of Artisan those days. Anything specific to be done concerning the flatpack distribution?

[wvengen]

I've worked on this issue, but am a bit stuck on creating a reproducible build environment for each platform (and legacy or not), so that ideally, anyone can create a final build (including me, to test changes in the release scripts well).

Regarding a new release, I think we will keep using the existing approach for now. Happy to update the Flatpak when a new version is released. Thanks for the question!

[MAKOMO]

Would be good if everybody could create builds. I agree. Dave, myself and you should work together on this. Maybe you can indicate where it hanged and maybe we have some ideas on how to resolve it.

Excellent! Thanks for supporting also our next release. I will update the changes in org.artisan_scope.artisan.metainfo.xml in one of my next commits.

I added you to our Cast. Hope you are fine with this.

[wvengen]

I'll dive into this again some time, and write down what I jumped into. It was mostly that I think the current build system has some prepared environments where the build is done. I could find Python version, but not other dependencies. To run a build locally, I needed to recreate each environment, somehow. That would either be Docker, or a virtualization solution like qemu or VirtualBox. Perhaps ideally, I'd like to be able to run the build in a Github Action, which would also allow others to fork and work on builds. But that was a larger chunk of work then I expected, so I put this down for a bit.

My first step was to experiment with creating a Python egg and wheels, and use that as the starting point to create an installer. That would have a clearly defined intermediate artifact between source code and final platform package, so that e.g. generating derived files (if needed) and copying data files can be done at a single place (instead of separately for each build script). If you would think that an good direction, I could starting preparing that as well. But adapting the build scripts would really benefit from a way to build and test locally, so perhaps that would be best to work on first.

Great you'll be updating the metainfo file, looking forward to releasing the next version on Flathub!

I feel honoured to be part of the Cast, thank you 🙏.

[roasterdave]

I can help with the environment dependencies if you want to develop another build process. We are always open to improvement. Building locally is attractive, but building on Appveyor is extremely easy. Anyone do it from a fork and there is no cost if the repository is public. The process is described here, How to Build Artisan. Appveyor also allows for hosting an instance on a local VM if you don't want to use theirs. If nothing else, the current build process is a useful reference point for a creating better method.

Welcome to the Cast!

[wvengen]

Thank you @roasterdave! I somehow missed the clear instructions to build, thanks for the pointer. I'll have a go at it 👍