Drydock is an opinionated tool offering a set of Tutor plugins aiming to provide features that enhance the operation of OpenedX installations in Kubernetes. It is developed by Edunext.
- A set of Kubernetes Jobs that replace the current tutor jobs with ArgoCD Sync Waves that allow for a more controlled deployment of openedx.
- A set of Kustomization overrides adding ArgoCD Sync Waves annotations to needed additional resources such as debug, workers or hpa.
- Backup cronjobs that allow backup of the MySQL and MongoDB databases.
- Integration of New Relic monitoring
- Add a custom nginx and cert-manager configuration
- Add a set of debug resources to help diagnose issues
Extra plugins added:
- Allows caddy to catch requests for multiple domains through
DRYDOCK_ENABLE_MULTITENANCY
- Add scorm matcher to caddy through
DRYDOCK_ENABLE_SCORM
- Patch for cms and lms worker pods to allow pod probes and lifecycle to work properly. Enabled via
DRYDOCK_POD_LIFECYCLE
You must install a supported release of this plugin to match the Open edX and Tutor version you are deploying. If you are installing this plugin from a branch in this Git repository, you must select the appropriate one:
Open edX release | Tutor version | Plugin branch | Plugin release |
---|---|---|---|
Maple | >=13.2, <14 |
Not supported | 0.7.x |
Nutmeg | >=14.0, <15 |
Not supported | 0.7.x |
Olive | >=15.0, <16 |
Not supported | 15.x.x |
Palm | >=16.0, <17 |
palm |
16.x.x |
Quince | >=17.0, <18 |
quince |
17.x.x |
Redwood | >=18.0, <19 |
main |
>=18.0.0 |
pip install tutor-contrib-drydock
tutor plugins enable drydock
tutor config save
tutor config save
The following configuration options are available:
DRYDOCK_INIT_JOBS
: Whether run the initialization jobs or not. Defaults tofalse
.DRYDOCK_CMS_SSO_USER
: The username of the CMS SSO user. Defaults tocms
.DRYDOCK_AUTO_TLS
: Whether to use cert-manager to automatically generate TLS certificates. Defaults totrue
.DRYDOCK_INGRESS
: Whether to deploy an ingress for the LMS and CMS. Defaults tofalse
.DRYDOCK_INGRESS_EXTRA_HOSTS
: A list of extra hosts to add to the ingress. Defaults to[]
.DRYDOCK_INGRESS_LMS_EXTRA_HOSTS
: A list of extra hosts to add to the LMS ingress. Defaults to[]
.DRYDOCK_CUSTOM_CERTS
: A dictionary of custom certificates to use with cert-manager. Defaults to{}
.DRYDOCK_NEWRELIC_LICENSE_KEY
: The New Relic license key. Defaults to""
.DRYDOCK_DEBUG
: Whether to deploy debug resources. Defaults tofalse
.DRYDOCK_ENABLE_MULTITENANCY
: Whether to enable multitennacy. Defaults totrue
.DRYDOCK_ENABLE_SCORM
: Whether to enable scorm. Defaults totrue
.DRYDOCK_POD_LIFECYCLE
: Whether to enable pod lifecycle. Defaults totrue
.NGINX_STATIC_CACHE_CONFIG
: A list of dictionaries with settings for different services to cache their assets in NGINX. The following is an example of the expected values:NGINX_STATIC_CACHE_CONFIG: {{service_name}}: host: {{service_host}} # e.g: {{LMS_HOST}} path: /static/ # you can specify a different path port: {{service_port}} # only needed if you have DRYDOCK_BYPASS_CADDY enabled
DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_MFE
: The minimum available percentage for the MFE's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_FORUM
: The minimum available percentage for the FORUM's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CADDY
: The minimum available percentage for the CADDY's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_LMS
: The minimum available percentage for the LMS's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_LMS_WORKER
: The minimum available percentage for the LMS WORKER's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CMS
: The minimum available percentage for the CMS's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CMS_WORKER
: The minimum available percentage for the worker's PodDisruptionBudget. To disable the PodDisruptionBudget, set0
. Defaults to0
.DRYDOCK_MIGRATE_FROM
: it allows defining the version of the OpenedX platform we are migrating from. It accepts the integer value mapping the origin release, for instance,13
(maple) or14
(nutmeg). When this variable is set, a group ofrelease-specific upgrade jobs
are added to the Kubernetes manifests. These jobs are applied to the cluster in a suitable order (thanks to the GitOps implementation with ArgoCD + sync waves) to guarantee the correct behavior of the platform in the new version. This brings thetutor k8s upgrade <https://github.com/overhangio/tutor/blob/v15.3.7/tutor/commands/k8s.py#L484>
_ command to the GitOps pattern. The release-specific upgrade jobs are supported from release13
(maple). Defaults to0
(which disables release-specific upgrade jobs)
NOTE: You also need to set
DRYDOCK_INIT_JOBS
totrue
to enable the release-specific upgrade jobs in the case of a platform migration.
Tutor doesn't generate manifest files for the initialization jobs, in consequence we can't use GitOps tools like ArgoCD to deploy the initialization jobs.
We had been using a static definition of the initialization jobs, but now we are using the Tutor filters <https://docs.tutor.edly.io/reference/api/hooks/filters.html>
_ to generate the kubernetes definition of the initialization jobs. This is a big improvement because now we can add new initialization jobs without modifying the Drydock code. The jobs are taken from COMMANDS_PRE_INIT
, COMMANDS_INIT
and CLI_DO_INIT_TASKS
Filters.
Tutor filter SYNC_WAVES_ORDER was added to allow define ArgoCD Sync Waves order and apply to the kubernetes resources through get_sync_waves_for_resource function.
We are defined by defult the following order:
All kubernetes resources
(except the ones that are defined in the next waves)Initialization Jobs
Upgrade Jobs
: When DRYDOCK_MIGRATE_FROM is set, over the Sync Wave 50CMS and LMS Deployments
: When DRYDOCK_POD_LIFECYCLE is active, over the Sync Wave 100Debug Resources
: When DRYDOCK_DEBUG active, over the Sync Wave 100Horizontal Pod Autoscalers
: When active, over the Sync Wave 150
NOTE: Quince uses Django 4.2 which only supports MySQL 8 or higher. You must upgrade your version of MySQL prior to performing the upgrade.
The upgrade from Maple to Palm fails because an issue with a squashed migration in edx-enterprise. To go around this issue, we need to apply migrations using an older version of edx-enterprise (3.60.4).
-
Run the sync to Palm without enabling the init jobs or upgrade jobs.
-
Once the LMS Deployment is running in the Palm version, go inside a pod and run the following:
pip install edx-enterprise==3.60.4 ./manage.py lms migrate pip install edx-enterprise==3.61.11
-
Now, you can enable the init jobs and upgrade jobs and run the sync again.
This workaround references the Andrey's comment
This guide delineates the necessary steps for a seamless migration to Quince using Drydock, ensuring a smooth transition with careful consideration of potential challenges.
Since Drydock version 17.3.0, a new mechanism was introduced to automatically generate Kubernetes manifest files for the initialization jobs defined by Tutor and Tutor plugins. The generated files are meant to be used by ArgoCD for deployment. Previously you would need to write the manifest files for the initialization jobs manually if you wanted to use a tool like ArgoCD.
-
For Quince, it is necessary to update the version of Drydock to the latest version 17.x.x in the requirements.txt file of your environment, including:
git+https://github.com/edunext/[email protected]#egg=drydock==17.x.x
-
In the
config.yml
file, include variables that activate the initialization jobs and post-migration jobs:DRYDOCK_INIT_JOBS: True DRYDOCK_MIGRATE_FROM: <MAJOR_OF_TUTOR_VERSION>
Set
DRYDOCK_MIGRATE_FROM
to the integer value mapping the origin release, for instance,13
(Maple) or14
(Nutmeg). Please refer to the Drydock configuration reference for a full description. -
Re-generate the configuration by running:
tutor config save
-
Push the changes generated by previous step to the corresponding manifests repository.
-
In ArgoCD, locate the corresponding application, and sync all resources.
-
If all synchronization occurs without issues, set:
DRYDOCK_INIT_JOBS: False
remove
DRYDOCK_MIGRATE_FROM
from you config file and run:tutor config save
-
Push the changes and sync again in ArgoCD.
This project is proposed as a possible way of creating a community maintained reference for large openedx installation. Sometimes the needs for customization in large instances of openedx goes against the required simplicity in the configuration that the tutor project strives for. In those cases, the solution is to create a tutor plugin that allows for such advanced customization options.
This projects intends to fill that gap with a solution that should allow many community members to collaborate in one repo on the heavy toll that is the maintainance of openedx operations.
This software is licensed under the terms of the AGPLv3.