Performance testing and improvement

[bmtcril]

As we approach our v1 release, it is important that we develop and maintain methodologies for testing performance of various parts of the system and occasionally checking for regressions. This epic holds the high level tasks for managing that work.

Event delivery

We currently have several ways of delivering xAPI events to ClickHouse. For each of these we should document a methodology and reference configuration for testing throughput of events to ClickHouse before issues emerge (queues filling up, task sizes growing, delivery time lagging, etc).

We should be able to emulate traffic using tracking log replay of very large files with a batch size of 1 and adjust the sleep setting to the maximum the backend can handle. If the backend can take a 0 sleep loop, we should run additional processes until it breaks.

• Investigate using Ralph to transform generated xapi logs back to tracking logs #196

We should be careful to constrain the configurations to be roughly equivalent in resources / cost, to emulate a production environment for a mid-sized system, and to use the same version of Aspects for all tests.

• Benchmark performance for xAPI over Celery to Ralph #202
• Benchmark performance for xAPI over Redis bus to Ralph #203
• Benchmark performance for xAPI over Kafka bus to Ralph #204
• Benchmark performance for xAPI over Vector to ClickHouse #205

Template for reporting results:

Test system configuration:
- Tutor version
- Aspects version
- Environment specifications (local / k8s, CPU / Memory / Disk resources allocated)

Load generation specifications:
- Tool
- Exact script
- Any custom settings for things like sleep time and # of processes

Data captured for results:
- Length of run
- Sleep time / batch size
- We should have values for these every 10 seconds:
  - Latency of events in ClickHouse (now - most recent event)
  - Queue size (if applicable) ex: pending tasks in celery, pending stream size in redis, etc
  - Total events in CH
  - Query times for 2-3 ClickHouse reporting queries (as taken from Superset)

Query performance

On a load test dataset, check every reporting query we have (as captured from the "show SQL" in Superset), with and without any applicable filters to see how they perform. We should run the queries 5x each and capture the response times and number of rows returned. It should also be possible to capture the queries by browsing each chart, using different filters, then pulling the SQL from the ClickHouse logs.

We should be careful to capture the xapi-db-load configuration for generating the data so we can regenerate as necessary.

• Capture all queries in Course Dashboard #208
• Capture all queries in Learner Dashboard #210
• Capture all queries in Learner Group Dashboard #209
• Run Course Dashboard test and store results #211
• Run Learner Dashboard test and store results #213
• Run Learner Group Dashboard test and store results #212

Template for reporting results:

Test ClickHouse configuration
- local / k8s/CH Cloud, Altinity...
- hardware or config specs
- Total rows in ClickHouse

For each query
- Query short name (enrollments no filter, enrollments enrollment type filter)
- Raw query
- Duration
- Rows returned

[bmtcril]

All data collected for the first sets of load tests are in these two files:

load_test_stats_1.txt
load_test_runs_1.txt

The Superset dashboard I'm using along with associated datasets can be imported from this zip:

dashboard_export_20240404T203605.zip

[bmtcril]

What we've found in #202 and earlier tests is that insert performance exceeds our ability to generate events up to that ~55 / sec line. Once we have more production information from partners we can determine if we should test to a higher threshold, but for now I'm closing these tasks out.