Upgrade Ubuntu

[mkllnk]

New servers should have a supported OS for as long as possible.
The just released Ubuntu 18.04 will probably be supported till 2023.
Update: It's 2023 now and we should go straight to Ubuntu 22.
Update: Our Ansible stack doesn't support past Ubuntu 20. So for now, we will upgrade the oldest servers from v16 to 20.

• [Upgrade Ubuntu] uk_staging #938
• [Upgrade Ubuntu] ca_prod #940
• [Upgrade Ubuntu] uk_prod #946
• [Upgrade Ubuntu] de_prod #957
• [Upgrade Ubuntu] ie_prod #958
• [Upgrade Ubuntu] be_prod #959
• [Upgrade Ubuntu] au_staging #939
• au_prod #966
• [Upgrade Ubuntu] us_prod #965

[elf-pavlik]

I will try to run current playbook in container running bionic, i think it may just work.

[enricostano]

We need to check the priority of this and the current situation of all the instances that are gradually adopting ofn-install. I propose to do that before we dedicate any effort to this.

[myriamboure]

Ping @pacodelaluna

[sauloperez]

In Katuma we will soon migrate our old and small production server and will do it with 18.04. I'll keep you posted.

[sigmundpetersen]

AFAIK every managed instance is running on Ubuntu 18, closing

[RachL]

@sigmundpetersen UK and FR at least are on 16.

[sigmundpetersen]

Yes I was confusing 18 compatibility with actual implementation on the servers. Sorry

[Matt-Yorkley]

Current status:

openfoodnetwork.be:	16.04
openfoodnetwork.org.uk:	16.04
openfoodnetwork.ie:	16.04
openfoodnetwork.de:	16.04
coopcircuits.fr:	16.04
openfoodnetwork.ca:	16.04
-----------------------------
app.katuma.org:         18.04
openfoodnetwork.net:	18.04
openfoodnetwork.org.au:	18.04

[mkllnk]

Servers still running on Ubuntu 16:

• staging.openfoodnetwork.org.au
• staging.openfoodnetwork.org.uk
• openfoodnetwork.ca
• openfoodnetwork.org.uk
• openfoodnetwork.ie
• openfoodnetwork.de
• coopcircuits.fr

Normal updates are over now. Extended security updates will last till 2026.

[mkllnk]

Estimate

I actually did the last upgrade of the French server. I logged 6h40m for that task.

• Migrate FR prod on new server #825

We probably need a similar time for the first server upgrade because there are always little issues coming up. But then we are in a position to just execute scripts for each server.

• 7 hours first server
• 2 hours per server afterwards

We have 11 prod servers and 3 staging servers. The full estimate is 7 + 2 * 13 = 33 hours or 5 days.

[rioug]

Github action has started throwing deprecation warning :

Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-node@v3. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

From the github post :

Our plan is to transition all actions to run on Node 20 by Spring 2024.

Node upgrade is currently blocked by the server upgrade, see openfoodfoundation/openfoodnetwork#10846 (review)

One more reason to get moving on this.

[mkllnk]

Current status:

$ ansible all_staging -u openfoodnetwork -a "lsb_release -a" | grep -F -e '|' -e Description
staging.openfoodnetwork.org.au | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.7 LTS
staging.openfoodnetwork.org.uk | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.3 LTS
staging.coopcircuits.fr | CHANGED | rc=0 >>
Description:	Ubuntu 18.04.5 LTS

$ ansible all_prod -u openfoodnetwork -a "lsb_release -a" | grep -F -e '|' -e Description
openfoodnetwork.org.au | CHANGED | rc=0 >>
Description:	Ubuntu 18.04.4 LTS
openfoodnetwork.org.nz | CHANGED | rc=0 >>
Description:	Ubuntu 20.04.6 LTS
openfoodnetwork.in | CHANGED | rc=0 >>
Description:	Ubuntu 20.04.6 LTS
openfoodnetwork.ca | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.6 LTS
openfoodnetwork.net | CHANGED | rc=0 >>
Description:	Ubuntu 18.04.5 LTS
openfoodnetwork.org.uk | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.5 LTS
app.katuma.org | CHANGED | rc=0 >>
Description:	Ubuntu 18.04.4 LTS
openfoodnetwork.ie | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.6 LTS
openfoodnetwork.de | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.6 LTS
coopcircuits.fr | CHANGED | rc=0 >>
Description:	Ubuntu 20.04.5 LTS
csicsoka.org | CHANGED | rc=0 >>
Description:	Ubuntu 18.04.6 LTS
openfoodnetwork.be | CHANGED | rc=0 >>
Description:	Ubuntu 16.04.5 LTS

Priority is updating Ubuntu 16 servers:

1. staging.openfoodnetwork.org.au
2. staging.openfoodnetwork.org.uk
3. openfoodnetwork.ca
4. openfoodnetwork.org.uk
5. openfoodnetwork.ie
6. openfoodnetwork.de
7. openfoodnetwork.be

[dacook]

Note that during server upgrades, we will not need to install and configure collectd on any servers for Wormly monitoring. We plan to retire it in favour of New Relic.

[mkllnk]

Just trying out Ubuntu 20.24 and it has Python 3.12 which doesn't seem compatible with our current version of Ansible. We use a very old version of Ansible which may need updating first. I'm questioning again if Ansible is the right tool for us...

[rioug]

what would you use instead of Ansible ?

[mkllnk]

what would you use instead of Ansible ?

Maybe just a bash script? At Ceres Fair Food we use a Git post-receive hook for deployments and it's only 40 lines of code.

Provisioning is a little bit more complicated in our current setup and maybe Ansible is a better fit for that. Ansible is good at recognising state and skipping tasks that are already done. But that also means that we go through a lot of code just to skip it and it slows things down. You can select to run only a certain tag but I would find it easier just to run a certain bash script.

One upside of Ansible is that lots of roles work with many different operating systems or distributions. But we are locked to Ubuntu anyway and don't use that feature.

[dacook]

We decided to de-scope and upgrade only the oldest servers, to the most recent version of Ubuntu supported by ofn-install, which is Ubuntu 20 (which runs out of support next April)

We'll proceed in this order:

1. uk_staging (first because they are keen to change hosts)
2. au_staging
3. ca_prod
4. uk_prod
5. de_prod
6. ie_prod
7. be_prod

(IE and BE are last because we haven't confirmed yet if they have funded the work).

[RachL]

@dacook it's confirmed that we can upgrade IE and BE 💪

[dacook]

I've updated the order above. AU staging had a blocker, so I will proceed with updating CA (ready to go) and UK (they are keen to change hosting providers anyway).

After that I will check if the hosting arrangements for each remaining server allow for Ubuntu 20, and if not, discuss with the team about next steps.

[dacook]

@RachL FYI, IE have already upgraded their own server! #951