Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add upgrading and clustering docs, update port setting names #96

Merged
merged 1 commit into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/how-tos/clickhouse_cluster.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
.. clickhouse-cluster:

How To Run Aspects With ClickHouse Cluster
******************************************

ClickHouse clusters are an advanced way of running ClickHouse, but offer many benefits for large scale installations. Deciding whether to run in a clustered environment is a big decision with many cost and administrative impacts and should be carefully considered before launching a production environment. In most cases we expect Aspects to perform well without cluster scaling.

.. warning::
Please fully understand the nuances of ClickHouse replication before choosing this route! While we have done some testing to ensure basic functionality of Aspects in a clustered environment, it may be expected to fail in novel and spectacular ways until we have a fully tested release running under load.


It is strongly suggested that if you are considering clustering you use a service provider to manage the cluster for you. `ClickHouse Cloud <https://clickhouse.com/cloud>`_ and `Altinity <https://altinity.com/cloud-database/>`_ both offer ClickHouse as a service. Altinity also offers `management services <https://altinity.com/cloud-database/#anywhere>`_ for running ClickHouse in your own Kubernetes environment.

Configuring Aspects to run in clustered mode is simple:

- Follow the directions in :ref:`remote-clickhouse` to set up your ClickHouse connection
- Set the ``CLICKHOUSE_CLUSTER_NAME`` Tutor configuration variables as appropriate for your installation
- Save your Tutor configuration: ``tutor config save``

When you initialize your environment the following changes will happen from a single server install:

- All users, databases, tables, views, functions, etc. will be created with ``ON CLUSTER 'your-cluster-name'``, including dbt managed views.
- All tables will be created with the Replicated versions of their table engines (ex: ``ReplicatedReplacingMergeTree``)
- ClickHouse traffic will be split among your nodes, as described in `their documentation <https://clickhouse.com/docs/en/architecture/cluster-deployment>`_ .
2 changes: 2 additions & 0 deletions docs/how-tos/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ How-Tos
:maxdepth: 2
:caption: Content:

Upgrade Aspects <upgrade>
Changing the xAPI actor identifier <changing_actor_identifier>
Backfill old or missing data <backfill>
Superset extra assets <superset_extra_assets>
Expand All @@ -15,3 +16,4 @@ How-Tos
Clickhouse extra SQL <clickhouse_sql>
Connect to external Clickhouse database <remote_clickhouse>
Extending DBT <dbt_extensions>
Run Aspects in a ClickHouse cluster <clickhouse_cluster>
4 changes: 2 additions & 2 deletions docs/how-tos/remote_clickhouse.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ Clickhouse instance. To do so, set the following variables in your **config.yaml
CLICKHOUSE_SECURE_CONNECTION: true|false
CLICKHOUSE_ADMIN_USER: <clickhouse_admin_user>
CLICKHOUSE_ADMIN_PASSWORD: <clickhouse_admin_password>
CLICKHOUSE_CLIENT_PORT: <clickhouse_client_port> # default to 9000 | 9440 used by initialization tasks
CLICKHOUSE_PORT: <clickhouse_port> # default to 8123 | 8443 used by other services such as ralph and the lms
CLICKHOUSE_HOST_HTTP_PORT: <clickhouse_http_port> # defaults to 8123 or 8443 for secure TLS connections
CLICKHOUSE_HOST_NATIVE_PORT: <clickhouse_client_port> # default to 9000 | 9440 used by initialization tasks

Additionally, there are some other variables that are calculated based on the above variables, but
can be customized if needed:
Expand Down
15 changes: 15 additions & 0 deletions docs/how-tos/upgrade.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
.. upgrade-aspects:

How-to Upgrade Aspects
**********************

At least in the early phases of existence, Aspects is intended to have a faster upgrade cycle than Open edX named releases. It's expected that many operators will want to upgrade between releases to get features as they become available, but the upgrade process should be the same whether it's happening as part of a named release upgrade or in between.

As for any upgrade you should take a backup snapshot of your environment before beginning, especially make sure you have a recent backup of your ClickHouse database. Then follow these steps, which are basically the same as installation:

- Install the version you would like from tutor-contrib-aspects, or for the latest: ``pip install --upgrade tutor-contrib-aspects``
- Save your tutor configuration: ``tutor config save``
- Build your openedx image (in case there are upgrades to LMS/Studio plugins): ``tutor images build openedx --no-cache``
- Initialize Aspects to get the latest schema and reports, for a tutor local install: ``tutor local do init -l aspects``

In a case where the release has special instructions, such as when new xAPI transforms have been added and you may need to replay tracking logs, they will be included in the release announcement.