rewrite static website using jekyll

[maul-esel]

As discussed, this PR rewrites the static website.

• Instead of dynamically loading static content via javascript, the website now uses a static site generator (jekyll) to generate static HTML at build time.
• For the webinterface, javascript is (of course) still used to query the backend API and to dynamically render the returned results. This part is largely unchanged.
• The URLs of the new site look prettier 😉 -- e.g. tools/automizer instead of ?ui=tool&tool=automizer . Old URLs are redirected (using javascript) to the new URLs for backward compatibility with existing links.
• This PR does not attempt to change the look or content of the site. Any such changes should be handled in a separate PR once this is merged.

Of course there is always more to improve, but I would consider the rewrite to be pretty much complete. Please point out any mistakes and bugs if present.

TODO

• Rewrite to jekyll
• Before this is merged, I plan to update and extend the site documentation, to make future maintenance easier. Feel free to let me know if there are any particularly important aspects to be documented.
• Set up automated deployment -- via jenkins, or via github Actions?

[danieldietsch]

Before we can merge this we need a proof-of-concept deployment. At the very least we need the following.

• Instructions on how to generate a new site for Windows and Linux, including all necessary dependencies
• An integration into Maven (best) or into a separate website deployment script that builds back and frontend (not so nice, because we would need Maven anyways to build the backend).

Regarding the deployment TODO: we should not use Github actions, because we do not want to concern us with Github's billing policies. We should use our Jenkins.

[danieldietsch]

@maul-esel Also, the tool pages do not work, and it looks like the web interface does not work as well. At least when I use Jekyll locally.

[danieldietsch]

When using jekyll serve I get various ...site.webmanifest not found errors. Should I be concerned?

[maul-esel]

• @bahnwaerter had some ideas on how to integrate the build in maven. Maybe we can discuss them next week?

• I don't mind using jenkins, but I'll probably need some help :)
  Just for completeness:

  GitHub Actions usage is free for standard GitHub-hosted runners in public repositories

  (https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions)

• The broken tool pages should be fixed, thanks (I missed this because I only added the website/ prefix at the last minute)

• Regarding the "not found" errors: Does the last commit fix this?

[maul-esel]

it looks like the web interface does not work as well

From the screenshot, it looks like the problem may be that no tool is selected -- the URL should have a parameter like ?tool=automizer or similar.

• How did you get there? Is there a broken link somewhere?
• I've restored the previous behaviour in such cases: redirect to the home page.

[bahnwaerter]

@bahnwaerter had some ideas on how to integrate the build in maven. Maybe we can discuss them next week? [...]

A very simple integration would be possible with the maven-exec-plugin. This plugin can execute programs like jekyll as if you would use a command line. We could use that in an extra Maven profile for generating the static website artifacts in the Maven lifecycle. However, an automatic deployment afterwards will be a bit more difficult. But we can talk about that next week.

[danieldietsch]

@maul-esel @bahnwaerter

1. now the links and the webinterface works. For the latter, the issue was that I did not do cp js/webinterface/config.dist.js js/webinterface/config.js, so its probably expected ;)
2. I again needed to do bundle update to get 4f0fe87 -- is there a way one can do that directly for multiple platforms?
3. Jenkins vs Github: no worries, I am happy to talk about how Jenkins works. Wrt. GH actions: ok, it looks free for now, but I still don't like it. For example, I just spent some time trying to find out what standard runners are and if there are non-standard runners. So far, I did not succeed. I also don't like the idea of supporting both, Jenkins and Github actions. And the default runners (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources) look like they might be too weak to run an Ultimate nightly.
4. Maven integration with maven-exec-plugin: Yes, this would be possible, but we would gain nothing compared to just using a bash script. Perhaps we should go along the lines of https://sketchingdev.co.uk/blog/continuous-deployment-of-jekyll-website-with-jenkins.html and add a docker runner while we are at it. A second WebBackend only job could be started so that we can combine the deployment of website and backend in one pipeline.

[maul-esel]

@bahnwaerter and I looked into the automatic build:

• We got maven to build the site using the maven-exec-plugin. @bahnwaerter still wants to make some changes (e.g. make it possible to build only the website, rather than all plugins) and then commit the changes here.
• We should add a possibility to distinguish between development and production builds (to maven).
• For production builds / during deployment, we should also copy the correct config.js.
• We still have the problem that different platforms lead to different Gemfile.lock files -- @danieldietsch, @bahnwaerter and I each get different results. I'll try to find some time to look into that.
• We need a jenkins job to deploy the site.
  • Do we want to use the docker plugin for jenkins, and run the frontend build in a container?
• We want to convert .epf to .json for the settings files, as part of the build / deployment.
  • While we're at it, we might also separate the tool settings from other settings in config.js.

[bahnwaerter]

@maul-esel can you undo your merge of the dev branch into this branch and do a rebase instead? Otherwise, I'm not able to review the code changes of this pull request.

[schuessf]

What exactly is needed here until we can merge it? There is now a python script that can convert settings (from epf to json). If I understand it correctly to build the website, you can just run this script and jekyll serve or am I missing something?

[danieldietsch]

@schuessf did you try that? How do we deploy it?

[maul-esel]

Let's try and revive this. I've looked through the changes again, added some more documentation, updated the dependencies, and migrated all recent changes to the website. From my side, this is ready to merge.

@danieldietsch You wanted to set up a docker runner on jenkins for this, right? To deploy, we need to execute the following steps:

cd trunk/source/WebsiteStatic/
./build_all_settings.py
bundle exec jekyll build --baseurl "/"
cp ../../../releaseScripts/website-config/frontend/config.js _site/js/webinterface/config.js
cp _site/ <wherever we currently deploy the files to>

[bahnwaerter]

So far, your changes look great. I tested to build the website using Jekyll 4.3.3 on Arch Linux and it works like charm. The diff of the Gemfile.lock looks as follows:

diff --git a/trunk/source/WebsiteStatic/Gemfile.lock b/trunk/source/WebsiteStatic/Gemfile.lock
index 8c1469c81e..2c8a9e0813 100644
--- a/trunk/source/WebsiteStatic/Gemfile.lock
+++ b/trunk/source/WebsiteStatic/Gemfile.lock
@@ -3,7 +3,6 @@ GEM
   specs:
     addressable (2.8.7)
       public_suffix (>= 2.0.2, < 7.0)
-    bigdecimal (3.1.8)
     colorator (1.1.0)
     concurrent-ruby (1.3.4)
     em-websocket (0.5.3)
@@ -11,14 +10,9 @@ GEM
       http_parser.rb (~> 0)
     eventmachine (1.2.7)
     ffi (1.17.0-x64-mingw-ucrt)
+    ffi (1.17.0-x86_64-linux-gnu)
     ffi (1.17.0-x86_64-linux-musl)
     forwardable-extended (2.6.0)
-    google-protobuf (4.27.3-x64-mingw-ucrt)
-      bigdecimal
-      rake (>= 13)
-    google-protobuf (4.27.3-x86_64-linux)
-      bigdecimal
-      rake (>= 13)
     http_parser.rb (0.8.0)
     i18n (1.14.5)
       concurrent-ruby (~> 1.0)
@@ -38,8 +32,8 @@ GEM
       safe_yaml (~> 1.0)
       terminal-table (>= 1.8, < 4.0)
       webrick (~> 1.7)
-    jekyll-sass-converter (3.0.0)
-      sass-embedded (~> 1.54)
+    jekyll-sass-converter (2.2.0)
+      sassc (> 2.0.1, < 3.0)
     jekyll-watch (2.2.1)
       listen (~> 3.0)
     kramdown (2.4.0)
@@ -54,7 +48,6 @@ GEM
     pathutil (0.16.2)
       forwardable-extended (~> 2.6)
     public_suffix (6.0.1)
-    rake (13.2.1)
     rb-fsevent (0.11.2)
     rb-inotify (0.11.1)
       ffi (~> 1.0)
@@ -62,9 +55,8 @@ GEM
       strscan
     rouge (4.3.0)
     safe_yaml (1.0.5)
-    sass-embedded (1.77.8)
-      google-protobuf (~> 4.26)
-      rake (>= 13)
+    sassc (2.4.0)
+      ffi (~> 1.9)
     strscan (3.1.0)
     terminal-table (3.0.2)
       unicode-display_width (>= 1.1.1, < 3)
@@ -79,6 +71,7 @@ GEM
 
 PLATFORMS
   x64-mingw-ucrt
+  x86_64-linux
   x86_64-linux-musl
 
 DEPENDENCIES

[maul-esel]

Thanks for trying it out @bahnwaerter!

I've added the missing platform to Gemfile.lock. My understanding is that, if you reset it to the committed version and try again, it should now remain unchanged. Could you try that? (though honestly I'm surprised at the amount of changes...)

[bahnwaerter]

My understanding is that, if you reset it to the committed version and try again, it should now remain unchanged. Could you try that?

I used your committed version and tried it again. Now I can confirm that it does not change any dependency in the Gemfile.lock. I actually don't know why there were so many changes, to be honest. But now it seems to work for me as expected.

[schuessf]

Thank you for your work!
I have checked the Jekyll Build on Windows and everything worked as expected.
I added some comments about the settings and build process.

[maul-esel]

OK, I've deleted the code_examples folder and re-purposed existing scripts to copy the examples during the build process. I've also simplified things: to build the website from scratch (including settings, code examples, and jekyll), one now only has to execute one script:

./scripts/build.py

For deployment (@danieldietsch), the following should now suffice:

cd trunk/source/WebsiteStatic/
./scripts/build.py --production
cp _site/ <wherever we currently deploy the files to>

(Note that Ultimate needs to be on the PATH during this call).

[danieldietsch]

The new Jenkins job "Deploy website" (for this PR: https://jenkins.sopranium.de/job/Ultimate/job/Deploy%20Website/view/change-requests/job/PR-634/) allows you to deploy the website to ultimate-pa.org. It looks like it works, but the site itself is missing examples. Any idea why? Is there something missing from building the website?

[maul-esel]

The relative path to the examples/ folder seems to have been off by one level, should be fixed now.