Running ASV Benchmarks

What are ASV Benchmarks and how do they work?

ASV is a benchmarking tool that is used by many prominent Python projects to benchmark and compare the performance of the library over time. Some prominent uses are by Numpy, Arrow, SciPy.

The tool has out-of-the-box support for creating benchmarks and using them to measure the performance over time (e.g. per commit, release, etc.) This is done by checking out, building and benchmarking each version. And the information is used to create graphics of the performance of the various versions on the various benchmarks. The latest benchmarks on the master versions can be seen here.

The benchmarks get run automatically in the following cases:

• nighty on the master branch - this updates the performance graphs
• on push on any branch with open PR - this benchmarks the branch in PR against the master branch and if there is a regression of more than 15% the benchmarks fail

Normally, ASV keeps track of the results in JSON files, but we are transforming them into data frames and store them in an ArcticDB database. There is a special script that helps in saving/extracting the jsons from the database.

Adding new benchmarks

All of the code for the actual benchmarks is located in the benchmarks folder. Any new benchmarks should be added there, either in one of the existing classes/files or in a new one. We are mainly using tests that benchmark the runtime (time_...) or peak memory usage (peakmem_...). But ASV support other benchmark type, you can read more about them in their docs.

Currently, we have the following 4 major groups of benchmarks:

1. Basic functions - for benchmarking operations such as read/write/append/update, their batch variant, etc. against a local storage
2. List functions - for benchmarking operations such as listing symbols, versions, etc. against a local storage
3. Local query builder - for benchmarking QB against a local storage (e.g. LMDB)
4. Persistent Query Builder - for benchmarking QB against a persistent storage (e.g. AWS S3) so we can read bigger data size
5. Resample - benchmarking resampling functionality. Parametrized over input rows, output rows, column types, and supported aggregators

It is important to understand how the each benchmark run is set up and torn down:

• setup_cache - is called only once, so any heavy computation should go in here (e.g. prepopulating some results in the DB)
• setup - is called before each benchmark run, so any setup that is light on computation should go in here (e.g. initializing the Arctic client)
• teardown - is the opposite of setup and is called after each benchmark run, so any cleanup that should be performed should go in here (be careful that you don't cleanup a library/symbol that is needed by a benchmark)

If you have made any changes to the benchmarks, you need to run them locally at least once and push the changed benchmarks.json file to GitHub. This file is very important for ASV and the results of the benchmarks will not be generated properly without it.

Running the benchmarks on master

There is a workflow that automatically benchmarks the latest master commit every night. But if you need to run it manually for some reason, you can issue a manual build from here and click on the Run workflow menu. This will start a build that will benchmark only the latest version.

If you have made changes to the benchmarks, you might need to regenerate all of the benchmarks. You will need to start a new manually build on master and select the run_all_benchmarks option.

Running the benchmarks on a non-master branch

Local Run

To run ASV locally, you first need to make sure that you have some prerequisites installed(both can be installed with pip), namely:

• asv
• virtualenv

You also need to change the asv.conf.json file to point to your branch instead on master (e.g. "branches": ["some_brnach"], ) If you have introduced any new hard dependencies, you need to add them to the matrix of dependencies that will be installed.

After that you can simply run:

python -m asv run -v --show-stderr HEAD^! => if you want to benchmark only the latest commit

To run a subset of benchmarks, use --bench <regex>.

After running this once, if you are just changing the benchmarks, and not ArcticDB code itself, you can run the updated benchmarks without committing, and therefore rebuilding again with:

python3 -m asv run --python=python/.asv/env/<some hash>/bin/python -v --show-stderr

where the path should be obvious from the first ASV run from HEAD^!

The benchmarks take between ~90 minutes per commit to run, depending on the machine. After the benchmarks have ran successfully, you can view the result as output in the console. You cannot generate the results as an html in a non-master branch, as this is a limitation of ASV.

GitHub Actions Run

If you want to benchmark more than one commit(e.g. if you have added new benchmarks), it might be better to run them on a GH Runner instead of locally. You will again need to change the asv.conf.json file to point to your branch instead on master (e.g. "branches": ["some_branch"], ) And if you have introduced any new hard dependencies, you need to add them to the matrix of dependencies that will be installed.

Then push your changes and start a manual build from here. Make sure to select your branch and whether or not you want to run the benchmarks against all commits.