Test: automated performance testing suite

[krizhanovsky]

Scope

Need to develop performance testing suite for

• HTTP/2 Web cache with 10 and 100 streams
• HTTPS Web cache
• HTTP/2 proxy mode with 10 and 100 streams
• HTTPS proxy mode

All the tests above must compare Tempesta FW against:

• optimized HAproxy
• optimized Nginx
• optimized Envoy (no cache)
• optimized Varnish
• previous results for Temesta FW 0.6.8 for now (no HTTP/2 tests so far)

The tests must measure:

• RPS for empty 200 responses, 1KB and 100KB responses
• avg and 99p latency (from Performance predictability test #1096) for empty 200 responses, 1KB and 100KB responses

These tests must run in two environments:

• KVM
• bare metal

The tests must run periodically in smoke (short) mode on CI and full run, including other web servers comparisons.

The test results should be stored in a server filesystem along with the configuration and system statistics (memory and CPU usage at first). A benchmark results must be also stored as text files with the command line to run the benchmark.

The CI jobs for the smoke performance tests must plot a Grafana graph to compare with previous runs and observe the trend.

Representing performance measurements

The benchmark runs must be cleaned to avoid results deviations. Different resources use 3-25 runs to get clean data and use different approaches for cleaning:

• Minimum values (Netflix)
• Statistical thresholds
• Student and Welch t-tests
• Many others

See https://bencher.dev/docs/explanation/thresholds/

References

Following issues address the problems, which must be revealed with the test suite, but require manual work.

• Non-keepalive HTTP requests handling #1415
• TDBv0.2: Cache background revalidation and eviction #515
• TLS: further performance improvements and cleanups #1064 (using tls-perf)

https://github.com/nyrkio/dsi - automated performance regression testing in Python, inherited from MongoDB

https://github.com/bencherdev/bencher - similar project in Rust

[ykargin]

Performance Testing Plan

1. Existing Stress Tests:
    Can be used for performance testing.
    Need to write a configuration with a reasonable number of requests and parameters.

2. Grafana for Results Visualization:
    Determine how to calculate metrics for each test.
    Initially, it is sufficient to have a single metric for each test (total of 4 metrics).

3. CI (Continuous Integration):
    Set up a dedicated worker (virtual machine) for running performance tests.
    Create a separate pipeline for execution. It should operate only on the dedicated virtual machine for performance testing.

4. Reporting Script:
    Develop a script that will:
        - Report results.
        - Log installed packages.
        - Store all information locally in an archived format.

5. Grafana Charts:
    Draw a separate chart in Grafana for each of the 4 test suites.

6. Running Tests against HAproxy/nginx/Envoy:
    Add the execution of these tests against HAproxy/nginx/Envoy.
    Display the results on the charts of the corresponding test suites.

[krizhanovsky]

We agreed on the call, that we'll go with adjusting our existing code from https://github.com/tempesta-tech/tempesta-test/ to build the performance regression test suite