[Upgrade Ubuntu] ca_prod

[dacook]

Checklist based on https://github.com/openfoodfoundation/ofn-install/wiki/Migrating-a-Production-Server

Setup

• check old server config for any additional services, eg /etc/nginx/sites-available
• hosting: provision new server with Ubuntu 20
  • and add DNS for new domain (prod2.openfoodnetwork.ca)
• Review host_vars config.yml
  • remove old users_sysadmin if needed
  • allow default postgres12
• Review ofn-secrets secrets.yml, remove unused config
  • Move to shared bugsnag
• add to inventory/hosts, copy temporary host_vars
• letsencrypt_proxy.yml
• setup.yml
• provision.yml
• deploy.yml
• transfer_assets.yml
• manually copy terms of service file (this probably wasn't necessary after
• db_integrations
• Check:
  • test reboot
  • send test mail (https://prod2.openfoodnetwork.ca/admin/mail_methods/edit).
  • sidekiq is NOT running yet (https://prod2.openfoodnetwork.ca/admin/sidekiq/metrics or sidekiq.log)
  • terms of service file: https://prod2.openfoodnetwork.ca/admin/terms_of_service_files
  • shop catalogue display correctly, with images, add to cart, begin checkout
• Check integrations
  • payment (check Stripe connect status https://prod2.openfoodnetwork.ca/admin/stripe_connect_settings/edit)
  • New Relic working, stats appearing
  • bugsnag
  • any others?

Migration

• deploy.yml matching version
• transfer_assets.yml
• old server: make a tiny data change to verify later
• old server: 🚧 maintenance_mode.yml
• old server: stop sidekiq
• db_transfer.yml
• transfer_assets.yml again, because ca_prod stores images locally (s3_access_key: not set).
• old server: stop puma, postgres (ensure other integrations no longer touch it)
• old server: shutdown redis-jobs. copy /var/lib/redis-jobs/dump.rdb to new server
• new server: systemctl restart puma; start sidekiq; start redis-jobs
• Rails.cache.clear (or migrate redis-cache/dump.rdb also)
• ⏭️ temporary_proxy.yml redirect traffic to new prod
• check everything is working
  • get confirmation from local team (eg David T or Theresa)
  • tiny data change is present. undo it.
  • shopfront and checkout looks good
  • sidekiq is running
  • n8n https://n8n.openfoodnetwork.org/workflow/QHqgDXTZEg2Wfiw0
  • https://waterlooregionfood.ca/directory/
• Update DNS to point to new server
• Update DNS for waterlooregionfood.ca also

Cleanup (after 48hrs)

• check server access logs to verify no traffic (or NR network graph)
• shut down the old server, cancel old DO droplet
• remove DNS for temporary subdomain
• make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.
• validate that provision.yml still works. This will rename ca-prod2 to ca-prod
• check metabase sync: https://data.openfoodnetwork.org.uk/admin/databases/ Not required
• n8n: CA - Garden Party Flower Plugs Orders Aggregation
• clean up local ofn-install/backups
• Update documentation:
  • https://github.com/openfoodfoundation/ofn-install/wiki/Current-servers

Rollback plan

• If an error occurs before the temporary proxy is active, and can't be resolved quickly, then restore service back to current server
• If an error occurs after proxy is active, users may have interacted with the new server (eg made payments.
  • if serious, consider putting into maintenance mode (and stop sidekiq) to avoid further changes
  • otherwise seek to resolve issue in-place.

Follow up

• Backups: there are none automated!
  • database is backed up and kept for last 4 releases (approx 1 month). but not stored offsite.
  • all asset files (incl product images) are not backed up https://github.com/openfoodfoundation/ofn-install/blob/master/roles/migrate_assets/defaults/main.yml#L8-L11
  • old snapshot in DO: confirm and destroy to save wasted $1.48 month

[dacook]

Hi @tschumilas and @David-OFN-CA , I'm ready to provision a new server for your upgrade. I've found I can choose the same options as the current one on Digital Ocean, at the same price. Before I kick off, I want to confirm with you first:

New droplet:

• Based in Toronto
• Ubuntu 20 (as per our current upgrade plan)
• Regular CPU and SSD plan
• 8gb RAM / 4 CPUs (same as before)
• 160 gb SSD (twice as much as before)

$48.00/month (same as before)

I hope to be able to complete the setup within a day, but would then need to perform the switch-over to the new server, which would need to be co-ordinated. It should be a seamless process, but I would choose a time that has least impact just in case. So there would be a time where there are two servers being paid for (perhaps 1-2weeks).

Regarding timing for the switchover, I would suggest 10pm EDT Sunday, which is 1pm my time and gives me time to resolve any issues. Can you let me know if this is ok, or if you'd prefer any other time of day or week?

[dacook]

All done!
Go-live occurred around 2024-10-31 14:45 (AEDT)
https://openfoodnetwork.slack.com/archives/CG7NJ966B/p1730346315048729?thread_ts=1727068302.322209&cid=CG7NJ966B

Including the Waterloo api cache service.

[dacook]

Actually we need to update the DNS for waterloo. waiting on access