Benchmark performance for xAPI over Vector to ClickHouse

[bmtcril]

No description provided.

[bmtcril]

Test 1 (4ec7bd) - 1k rows

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: true
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: False
  • ASPECTS_VECTOR_STORE_TRACKING_LOGS: false
  • ASPECTS_VECTOR_STORE_XAPI: true

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend vector
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags vector 1000 local nobatch novectortracking

Data captured for results:

• Length of run
  • Event generation: 0:00:27.025545.
  • Monitoring / total run length: 0:00:40.183303
• Sleep time: 0
• Events: 1000
• Raw stats attached: 4ec7bd_stats.txt

Findings:

• Vector was able to keep up, ClickHouse was 1-5 seconds behind the entire 30 second test
• The Vector max queue was 31, though this can be misleading compared to other backends since it only represents the number of read log lines vs. sent log lines, and does not count log lines not yet read. That said, there was no indication of back pressure in this test that would invalidate this number.
• Inserted rows per second peaked at 52.4, but the test was fast. The 10k test should show better sustained performance.

[bmtcril]

Test 2 (9a6da6) - 10k rows

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: true
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: False
  • ASPECTS_VECTOR_STORE_TRACKING_LOGS: false
  • ASPECTS_VECTOR_STORE_XAPI: true

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend vector
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 10000 --sleep_time 0 --tags vector 10k local nobatch novectortracking

Data captured for results:

• Length of run
  • Event generation: 0:03:40.012385
  • Monitoring / total run length: 0:03:52.397353
• Sleep time: 0
• Events: 10000
• Raw stats attached: 9a6da6_stats.txt

Findings:

• Vector was able to keep up, ClickHouse was 0-2 seconds behind the entire 220 second test
• The Vector max queue was 60, though this can be misleading compared to other backends since it only represents the number of read log lines vs. sent log lines, and does not count log lines not yet read. That said, there was no indication of back pressure in this test that would invalidate this number.
• Inserted rows per second peaked at 55, most of the test was consistently in the 40s

[bmtcril]

What we've found in #202 and earlier tests is that insert performance exceeds our ability to generate events up to that ~55 / sec line. Once we have more production information from partners we can determine if we should test to a higher threshold, but for now I'm closing these tasks out.