Skip to content

Latest commit

ย 

History

History
375 lines (317 loc) ยท 26.5 KB

README.md

File metadata and controls

375 lines (317 loc) ยท 26.5 KB

๐Ÿฐ Bencher Bencher

Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they hit production.

  • Run: Run your benchmarks locally or in CI using your favorite benchmarking tools. The bencher CLI simply wraps your existing benchmark harness and stores its results.
  • Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
  • Catch: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!


Bencher consists of:

  • bencher CLI
  • Bencher API Server
  • Bencher Console Web UI

Though Bencher is open source, there is also a hosted version available Bencher Cloud.

The best place to start is the Bencher Quick Start tutorial.


Start Continuous Benchmarking

๐Ÿฐ Use the GitHub Action with your project

Documentation

๐ŸŒ Also available in:

Supported Benchmark Harnesses

๐Ÿ‘‰ For more details see the explanation of benchmark harness adapters.

Showcase

Microsoft

Microsoft CCF

Rustls TLS Library

Rustls

Diesel

Diesel

Hydra Database

Hydra Database

GreptimeDB

GreptimeDB

Tailcall

Tailcall

Ratatui

Ratatui

Poolifier

Poolifier

Wire

Wire

๐Ÿ‘‰ Checkout all public projects.

GitHub Actions

Install the Bencher CLI using the GitHub Action, and use it for continuous benchmarking in your project.

name: Continuous Benchmarking with Bencher
on:
  push:
    branches: main
jobs:
  benchmark_with_bencher:
    name: Benchmark with Bencher
    runs-on: ubuntu-latest
    env:
      BENCHER_PROJECT: my-project-slug
      BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: bencherdev/bencher@main
      - run: bencher run "bencher mock"

Supported Operating Systems:

  • Linux (x86_64 & ARM64)
  • MacOS (x86_64 & ARM64)
  • Windows (x86_64 & ARM64)

๐Ÿ‘‰ For more details see the explanation of how to use GitHub Actions.

Repository Secrets

Add BENCHER_API_TOKEN to you Repository secrets (ex: Repo -> Settings -> Secrets and variables -> Actions -> New repository secret). You can find your API tokens by running bencher token list my-user-slug or view them in the Bencher Console.

Error on Alert

You can set the bencher run CLI subcommand to error if an Alert is generated with the --err flag.

bencher run --err "bencher mock"

๐Ÿ‘‰ For more details see the explanation of bencher run.

Comment on PRs

You can set the bencher run CLI subcommand to comment on a PR with the --github-actions argument.

bencher run --github-actions "${{ secrets.GITHUB_TOKEN }}" "bencher mock"

๐Ÿ‘‰ For more details see the explanation of bencher run.

Example PR Comment


Branch254/merge
Testbedubuntu-latest
๐Ÿšจ 1 ALERT: Threshold Boundary Limit exceeded!
BenchmarkMeasure
Units
ViewBenchmark Result
(Result ฮ”%)
Lower Boundary
(Limit %)
Upper Boundary
(Limit %)
Adapter::JsonLatency
nanoseconds (ns)
๐Ÿ“ˆ plot
๐Ÿšจ alert
๐Ÿšท threshold
3,445.60
(+1.52%)
3,362.07
(102.48%)
Click to view all benchmark results
BenchmarkLatencyBenchmark Results
nanoseconds (ns)
(Result ฮ”%)
Upper Boundary
nanoseconds (ns)
(Limit %)
Adapter::Json๐Ÿ“ˆ view plot
๐Ÿšจ view alert
๐Ÿšท view threshold
3,445.60
(+1.52%)
3,362.07
(102.48%)
Adapter::Magic (JSON)๐Ÿ“ˆ view plot
๐Ÿšท view threshold
3,431.40
(+0.69%)
3,596.95
(95.40%)
Adapter::Magic (Rust)๐Ÿ“ˆ view plot
๐Ÿšท view threshold
22,095.00
(-0.83%)
24,732.80
(89.33%)
Adapter::Rust๐Ÿ“ˆ view plot
๐Ÿšท view threshold
2,305.70
(-2.76%)
2,500.49
(92.21%)
Adapter::RustBench๐Ÿ“ˆ view plot
๐Ÿšท view threshold
2,299.90
(-3.11%)
2,503.41
(91.87%)
๐Ÿฐ View full continuous benchmarking report in Bencher

Specify CLI Version

There is also an optional version argument to specify an exact version of the Bencher CLI to use. Otherwise, it will default to using the latest CLI version.

- uses: bencherdev/bencher@main
  with:
    version: 0.4.20

Specify an exact version if using Bencher Self-Hosted. Do not specify an exact version if using Bencher Cloud as there are still occasional breaking changes.

Share Your Benchmarks

All public projects have their own perf page. These results can easily be shared with an auto-updating perf image. Perfect for your README!

Benchmark Adapter Comparison for Bencher - Bencher

Contributing

The easiest way to contribute is to open this repo as a Dev Container in VSCode by simply clicking one of the buttons below. Everything you need will already be there! Once set up, both the UI and API should be built, running, and seeded at localhost:3000 and localhost:61016 respectively. To make any changes to the UI or API though, you will have to exit the startup process and restart the UI and API yourself.

๐Ÿฐ All pull requests should target the devel branch


Bencher VSCode Dev Container

Bencher GitHub Codespaces

There is also a pre-built image from CI available for each branch: ghcr.io/bencherdev/bencher-dev-container

License

All content that resides under any directory or feature named "plus" is licensed under the Bencher Plus License.

All other content is licensed under the Apache License, Version 2.0 or MIT License at your discretion.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Bencher by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.