🔬 Regression Tests for ClickHouse®

This directory contains integration tests written using 👽 TestFlows 🛸 that involves several ClickHouse instances, custom configs, ZooKeeper, etc.

🗓 Scheduled Regression Runs

Results for the latest scheduled workflow runs.

ClickHouse Version	Status
`head`
`latest`
`24.8`
`24.3`
`23.8`
`23.3`
`24.3.12.76.altinitystable`
`23.8.11.29.altinitystable`
`23.3.19.33.altinitystable`
`22.8.15.25.altinitystable`
`22.3.15.34.altinitystable`

📅 Timetable of Scheduled Regression Runs

Day	latest & head	Every Other Version
Sunday	❌ No	❌ No
Monday	❌ No	❌ No
Tuesday	✅ Yes	❌ No
Wednesday	❌ No	❌ No
Thursday	✅ Yes	❌ No
Friday	❌ No	❌ No
Saturday	✅ Yes	✅ Yes

🔍 Table of Contents

1 Supported Environment
2 Prerequisites
3 Running CI/CD
- 3.1 Running Only Specific Suites
- 3.2 Running From Docker Image
- 3.3 Running From Altinity Repo
- 3.4 Running From ClickHouse PR
4 Running CI/CD With CI/CD Trigger
5 CI/CD Secrets and Variables
- 5.1 Variables
- 5.2 Secrets
6 Running Tests Locally
- 6.1 Output Verbosity
- 6.2 Running Only Selected Tests
- 6.3 How To Debug Why Test Failed
  - 6.3.1 Step 1: Find Which Tests Failed
  - 6.3.2 Step 2: Download test.log That Contains All Raw Messages
  - 6.3.3 Step 3: Get Messages for the Failing Test
  - 6.3.4 Step 4: Working With the test.log
- 6.4 Running S3 Suites
  - 6.4.1 Minio
  - 6.4.2 AWS S3 Storage
  - 6.4.3 GCS (Google Cloud Storage)
- 6.5 Running Tiered Storage Suites
  - 6.5.1 Normal
  - 6.5.2 Running on Minio
  - 6.5.3 Running on AWS S3 Storage
  - 6.5.4 Running on GCS Storage
  - 6.5.5 Pausing In Tests
7 Running GitHub Actions

Supported Environment

Ubuntu 22.04 / 24.04
Python 3 >= 3.10.12
Docker >= Docker version 25.0.3, build 4debf41
Docker Compose >= v2.23.1 (non-Python version)

Prerequisites

Docker

Standalone `docker-compose`

Standalone docker-compose binary.

Docker Compose

For example,

x86_64

sudo curl -SL https://github.com/docker/compose/releases/download/v2.23.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Aarch64 (ARM64)

sudo curl -SL https://github.com/docker/compose/releases/download/v2.23.1/docker-compose-linux-aarch64 -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Python modules

To install all necessary Python modules (including TestFlows), execute the following command:

pip3 install -r requirements.txt

Running CI/CD

Running Only Specific Suites

Specify suite variable to select running only specific suites

Variable	Name	Description
`suite`	`aes_encryption`	AES Encryption suite
`suite`	`aggregate_functions`	Aggregate Functions suite
`suite`	`atomic_insert`	Atomic Insert suite
`suite`	`alter_all`	Full Alter suite
`suite`	`alter_replace_partition`	Alter Replace Partition suite
`suite`	`alter_attach_partition`	Alter Replace Attach suite
`suite`	`alter_move_partition`	Alter Move Attach suite
`suite`	`base_58`	Base58 suite
`suite`	`benchmark_all`	All Ontime Benchmark (MinIO, AWS, GCS) suite
`suite`	`benchmark_aws`	Ontime Benchmark AWS suite
`suite`	`benchmark_gcs`	Ontime Benchmark GCS suite
`suite`	`benchmark_minio`	Ontime Benchmark MinIO suite
`suite`	`clickhouse_keeper`	ClickHouse Keeper (No SSL and SSL FIPS) suite
`suite`	`data_types`	Data Types suite
`suite`	`datetime64_extended_range`	Extended DateTime64 suite
`suite`	`disk_level_encryption`	Disk Level Encryption
`suite`	`dns`	DNS suite
`suite`	`engines`	Engines suite
`suite`	`example`	Example suite
`suite`	`extended_precision_data_types`	Extended Precision Data Types suite
`suite`	`kafka`	Kafka suite
`suite`	`kerberos`	Kerberos suite
`suite`	`ldap`	LDAP suite
`suite`	`lightweight_delete`	Lightweight Delete suite
`suite`	`parquet_all`	Full Parquet Suite
`suite`	`parquet`	Parquet Data Type suite
`suite`	`parquet_minio`	Parquet MinIO suite
`suite`	`parquet_s3`	Parquet AWS S3 suite
`suite`	`part_moves_between_shards`	Part Moves Between Shards suite
`suite`	`rbac`	RBAC suite
`suite`	`s3_all`	All S3 (MinIO, AWS, GCS) suites
`suite`	`s3_aws`	S3 AWS suite
`suite`	`s3_gcs`	S3 GCS suite
`suite`	`s3_minio`	S3 MinIO suite
`suite`	`selects`	Selects suite
`suite`	`session_timezone`	Session Timezone suite
`suite`	`ssl_server`	SSL Server suite
`suite`	`tiered_storage_all`	All Tiered Storage (Local Disk, MinIO, AWS, GCS) suites
`suite`	`tiered_storage_aws`	Tiered Storage AWS suite
`suite`	`tiered_storage_gcs`	Tiered Storage GCS suite
`suite`	`tiered_storage_local`	Tiered Storage Local suite
`suite`	`tiered_storage_minio`	Tiered Storage MinIO suite
`suite`	`vfs_all`	Full VFS (MinIO, AWS, GCS) suite
`suite`	`vfs_aws`	VFS AWS suite
`suite`	`vfs_gcs`	VFS GCS suite
`suite`	`vfs_minio`	VFS MinIO suite
`suite`	`window_functions`	Window Functions suite

Running From Docker Image

When running the CI/CD pipeline, provide the following variables:
Example values using altinity/clickhouse-server:21.8.15.15.altinitystable

Variables
`Variable`	`package`	`docker://altinity/clickhouse-server`
`Variable`	`version`	`21.8.15.15.altinitystable`

Running From Altinity Repo

When running the CI/CD pipeline, provide the following variables:

Variables
`Variable`	`package`	`deb://builds.altinity.cloud/apt-repo/pool/main`
`Variable`	`version`	The version to use for tests. For example, `21.8.8.1.altinitystable`

Running From ClickHouse PR

Get the link to the deb package: PR -> ClickHouse build check (actions) -> Details -> copy link to a deb package.
Break down the link into CI/CD variables:
Example values using https://s3.amazonaws.com/clickhouse-builds/37882/f74618722585d507cf5fe6d9284cf32028c67716/package_release/clickhouse-client_22.7.1.1738_amd64.deb

Variables
`Variable`	`package`	`deb://s3.amazonaws.com/clickhouse-builds/37882/f74618722585d507cf5fe6d9284cf32028c67716/package_release` NOTE: 'deb' instead of 'https' and no '/' on the end.
`Variable`	`version`	`22.7.1.1738`
`Variable`	`package_version_postfix`	By default `all` (supports older versions), specify `amd64` for newer PRs where all packages have `amd64` postfix.

Running CI/CD With CI/CD Trigger

To run the CI/CD pipline you can use cicd-trigger.py ./cicd-trigger.py with following options

-w --wait             Wait for pipeline to finish.
--package             Specify docker:// or deb:// package.
--version             Specify clickhouse version.
--package-postfix     Postfix of the clickhouse-server and clickhouse-client deb package, default: 'amd64'. Choices 'amd64', 'all'.
--suite               Select test suite to run, default: 'all'. Choices "all", "aes_encryption", "aggregate_functions", "atomic_insert", "base_58", "clickhouse_keeper", "datetime64_extended_range", "disk_level_encryption", "dns", "example", "extended_precision_data_types", "kafka", "kerberos", "ldap", "lightweight_delete", "map_type", "parquet","parquet_minio", "parquet_s3", "part_moves_between_shards", "rbac", "s3", "s3_aws", "s3_gcs", "selects", "ssl_server", "tiered_storage", "tiered_storage_aws", "tiered_storage_gcs", "window_functions", "benchmark".
--output              Tests stdout output style, default: 'classic'. Choices 'nice', 'classic', 'short', etc.
--parallel            Enable or disable running tests in parallel, default: 'on'. Choices 'on', 'off'.
--token               Personal access token or private token with api access to the gitlab project, default: 'GITLAB_TOKEN' environment variable.
--options             Extra options that will be added to test run command.
--arch                Architecture to run the tests on, default: 'amd64'. Choices 'amd64', 'arm64'.
--branch              Specify which branch to run the tests on, default: 'main'.
--artifacts           Specify whether to upload artifacts internally or publically, default: 'internal'. Choices 'internal', 'public'. Bucket for internal upload: 'altinity-internal-test-reports'. Bucket for public upload 'altinity-test-reports'.
--debug               Enable script running in debug mode, default: 'False'. Choices 'True', 'False'.

CI/CD Secrets And Variables

Variables

The CI/CD has the following variables:

Variable	Purpose	Default
`PARALLEL`	Specify whether to run tests within each suite in parallel.	1
`UPLOAD_LOGS`	Specify whether to upload logs.	1

Secrets

The CI/CD has the following secrets:

Secret	Purpose	Notes
`AWS_ACCESS_KEY`	AWS Secret Access Key for the bucket used during testing.
`AWS_KEY_ID`	AWS Access Key ID for the bucket used during testing.
`AWS_BUCKET`	AWS Bucket used during testing.
`AWS_REGION`	AWS Region of the bucket used during testing.
`GCS_KEY_ID`	GCS Key ID for the bucket used during testing.
`GCS_KEY_SECRET`	GCS Key Secret for the bucket used during testing.
`GCS_URI`	GCS URI of the bucket used for testing.
`AWS_REPORT_KEY_ID`	AWS Access Key ID for the report bucket.	The report buckets are hardcoded in `create_and_upload_logs.sh`.
`AWS_REPORT_SECRET_ACCESS_KEY`	AWS Secret Access Key for the report bucket.	The report buckets are hardcoded in `create_and_upload_logs.sh`.
`AWS_REPORT_REGION`	AWS Region of the report bucket.	The report buckets are hardcoded in `create_and_upload_logs.sh`.
`DOCKER_USERNAME`	Docker username for login to prevent pull limit.	Not necessary if running small amount of tests.
`DOCKER_PASSWORD`	Docker password for login to prevent pull limit.	Not necessary if running small amount of tests.

Running Tests Locally

You can run tests locally by passing --local and --clickhouse to the top level regression.py or cd into any sub-folders to run suite-specific regression.py.

--local specifies that regression will be run locally
--clickhouse specifies a path to a ClickHouse package or binary that will be used during the regression run. You can also use a docker image using the docker:// prefix. For example, --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine
--base-os specifies a docker image to install ClickHouse into when --clickhouse is not a docker image. Supported distributions are altinityinfra/clickhouse-regression-multiarch:2.0 (default), ubuntu, and alpine. There is experimental support for fedora and redhat/ubi9.

Tip

You can pass the -h or --help argument to the regression.py to see a help message.

python3 regression.py -h

Important

Make sure that the ClickHouse binary has correct permissions. If you are using /usr/bin/clickhouse its owner and group are set to root:root by default and it needs to be changed to clickhouse:clickhouse. You can change the owner and the group using the following command.

sudo chown clickhouse:clickhouse /usr/bin/clickhouse

Note

If you need to restore the old behavior of copying binaries out of packages, instead of installing them, use the --as-binary flag.

Using the default ClickHouse installation and its server binary at /usr/bin/clickhouse, you can run regressions locally using the following command.

python3 regression.py --local --clickhouse "/usr/bin/clickhouse"

Output Verbosity

You can control the verbosity of the output by specifying the output format with the -o or --output option. See --help for more details.

Running Only Selected Tests

You can run only the selected tests by passing the --only option to the regression.py.

For example,

./regression.py --local --clickhouse /usr/bin/clickhouse --only "/clickhouse/rbac/syntax/grant privilege/*"

will execute all rbac/syntax/grant privilege tests.

If you want to run only a single test such as the /clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='KILL QUERY', on=('*.*',), allow_introspection=False you can do it as follows

./regression.py --local --clickhouse /usr/bin/clickhouse --only "/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='KILL QUERY', on=('[*].[*]',), allow_introspection=False/*"

Note

You need to surround special characters such as * with square brackets, for example [*].
You need to end the filtering pattern with /* to run all the steps inside the test.

For more information, please see the Filtering section in the TestFlows Handbook.

How To Debug Why Test Failed

Step 1: Find Which Tests Failed

If the TestFlows check does not pass, you should look at the end of the test_run.txt.out.log to find the list of failing tests. For example,

clickhouse_testflows_tests_volume
Start tests
➤ Dec 02,2020 22:22:24 /clickhouse
...
Failing

✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=('db0.table0', 'db0.*', '*.*', 'tb0', '*'), allow_column=True, allow_introspection=False
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege
✘ [ Fail ] /clickhouse/rbac/syntax
✘ [ Fail ] /clickhouse/rbac
✘ [ Fail ] /clickhouse

In this case, the failing test is

/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=('db0.table0', 'db0.*', '*.*', 'tb0', '*'), allow_column=True, allow_introspection=False

while the others

✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege
✘ [ Fail ] /clickhouse/rbac/syntax
✘ [ Fail ] /clickhouse/rbac
✘ [ Fail ] /clickhouse

failed because the first fail gets "bubble-up" the test execution tree all the way to the top-level test which is the /clickhouse.

Step 2: Download `test.log` That Contains All Raw Messages

You need to download the test.log that contains all raw messages.

Step 3: Get Messages For The Failing Test

Once you know the name of the failing test, and you have the test.log that contains all the raw messages for all the tests, you can use tfs show test messages command.

You get the tfs command by installing TestFlows.

For example,

cat test.log | tfs show test messages "/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=\('db0.table0', 'db0.\*', '\*.\*', 'tb0', '\*'\), allow_column=True, allow_introspection=False"

Note

Characters that are treated as special in extended regular expressions need to be escaped. In this case we have to escape the *, (, and the ) characters in the test name.

Step 4: Working With the `test.log`

You can use the test.log with many of the commands provided by the tfs utility.

See tfs --help for more information.

For example, you can get a list of failing tests from the test.log using the tfs show fails command as follows

cat test.log | tfs show fails

or get the results using the tfs show results command as follows

cat test.log | tfs show results

or you can transform the log to see only the new fails using the tfs transform fail --new command as follows

cat test.log | tfs transform fails --new

Running S3 Suites

Minio

Minio is the default test suite, but can be specified using --storage minio.

Examples:

Explicit storage declaration:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --storage minio

Utilizing default values:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine

You can also specify the minio uri (--minio-uri), root user (--minio-root-user), and root password (--minio-root-password). However, this is not necessary.

AWS S3 Storage

Aws requires a region(--aws-s3-region) and a bucket(--aws-s3-bucket) (the bucket must end with /), in addition to the key id and secret access key. Aws can be specified using --storage aws_s3.

Env variables:

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_DEFAULT_REGION=
export S3_AMAZON_BUCKET=

Examples:

Inline:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --aws_s3_key_id [masked] --aws_s3_access_key [masked] --aws-s3-bucket [masked] --aws-s3-region [masked] --storage aws_s3

Env:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --storage aws_s3

GCS (Google Cloud Storage)

GCS requires a gcs uri (--gcs-uri) (the uri must end with /), gcs key id (--gcs-key-id), and gcs key secret(--gcs-key-secret), in addition to the s3 key id and secret access key. GCS can be specified using --storage gcs.

Env variables:

export GCS_URI=
export GCS_KEY_ID=
export GCS_KEY_SECRET=

Examples:

Inline:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --gcs-uri [masked] --gcs-key-id [masked] --gcs-key-secret [masked] --storage gcs

Env:

s3/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --storage gcs

Running Tiered Storage Suites

Normal

Normal tiered storage suite does not require any variables to be provided.

From the regression directory, it can be run with the following command:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine

Running on Minio

Minio tiered storage suite only requires that --with-minio is specified.

It can be run with the following command:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --with-minio

Running on AWS S3 Storage

AWS S3 tiered storage requires an access key (--aws-s3-access-key), a key id (--aws-s3-key-id), and a uri (--aws-s3-uri). The uri must end with /. These can be passed as environment variables. AWS S3 must be specified using --with-s3amazon.

Env variables:

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export S3_AMAZON_URI=

Examples:

Inline:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --aws-s3-key-id [masked] --aws-s3-access-key [masked] --aws-s3-uri [masked] --with-s3amazon

Env:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --with-s3amazon

Running on GCS Storage

GCS tiered storage requires a gcs uri (--gcs-uri) (the uri must end with /), gcs key id (--gcs-key-id), and gcs key secret(--gcs-key-secret). GCS can be specified using --with-s3gcs.

Env variables:

export GCS_URI=
export GCS_KEY_ID=
export GCS_KEY_SECRET=

Examples:

Inline:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --gcs-uri [masked] --gcs-key-id [masked] --gcs-key-secret [masked] --with-s3gcs

Env:

tiered_storage/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:22.3.6.5-alpine --with-s3gcs

Pausing in Tests

You can explicitly specify PAUSE_BEFORE, PAUSE_AFTER, PAUSE_ON_PASS, and PAUSE_ON_FAIL flags inside your test program. For example,

with Test("my test"):
    with Step("my step 1", flags=PAUSE_BEFORE):
        note("my step 1")
    
    with Step("my step 2", flags=PAUSE_AFTER):
        note("my step 2")

    with Step("my step 2", flags=PAUSE_ON_PASS):
        note("my step 2")
        
    with Step("my step 2", flags=PAUSE_ON_FAIL):
        note("my step 2")

For decorated tests Flags decorator can be used to set these flags.

@TestScenario
@Flags(PAUSE_BEFORE|PAUSE_AFTER) # pause before and after this test
def my_scenario(self):
    pass

This can be used for getting access to the Docker Compose environment with conditions equal to cluster condition on the current step by executing standard Docker Compose commands ("ps", "exec" etc.) from the "*_env" folder. It allows us to make some manual checks/changes on dockers and continue test with new manually set conditions.

Running GitHub Actions

To run GitHub actions, navigate to Actions, and select Run CI/CD regression tests. Inside the Run workflow dropdown menu specify the package, version, suite, and upload destination of artifacts.

Package: docker:// or https:// package specifier to use for tests. For example:

docker://altinity/clickhouse-server
docker://clickhouse/clickhouse-server
https://s3.amazonaws.com/altinity-build-artifacts/217/acf34c9fc6932aaf9af69425612070b50529f484/package_release/clickhouse-client_22.8.11.17.altinitystable_amd64.deb

Version: Version of clickhouse to use for tests. The test verifies that the node version matches the specified version. When the package option uses the docker:// specifier then the version is the image tag. For example:

22.3.9.19-alpine
22.3.8.40.altinitystable
latest

Suite: Which suite to run. Default runs all suites.

Artifacts: Specify whether to upload to internal or public s3 bucket. 'altinity-internal-test-reports' for internal upload, 'altinity-test-reports' for public.

Name		Name	Last commit message	Last commit date
Latest commit History 5,749 Commits
.github		.github
aes_encryption		aes_encryption
aggregate_functions		aggregate_functions
alter		alter
atomic_insert		atomic_insert
attach		attach
base_58		base_58
clickhouse		clickhouse
clickhouse_keeper		clickhouse_keeper
clickhouse_keeper_failover		clickhouse_keeper_failover
container-images/test		container-images/test
data_types		data_types
datalakes		datalakes
datetime64_extended_range		datetime64_extended_range
disk_level_encryption		disk_level_encryption
dns		dns
docker-compose		docker-compose
docker		docker
docs		docs
engines		engines
example		example
extended_precision_data_types		extended_precision_data_types
functions		functions
helpers		helpers
image		image
integration		integration
jwt_authentication		jwt_authentication
kafka		kafka
kerberos		kerberos
key_value		key_value
ldap		ldap
lightweight_delete		lightweight_delete
memory		memory
ontime_benchmark		ontime_benchmark
parquet		parquet
part_moves_between_shards		part_moves_between_shards
rbac		rbac
s3		s3
selects		selects
session_timezone		session_timezone
ssl_keeper		ssl_keeper
ssl_server		ssl_server
tiered_storage		tiered_storage
window_functions		window_functions
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
LICENSE		LICENSE
README.md		README.md
altinity.png		altinity.png
cicd-trigger.py		cicd-trigger.py
clean		clean
regression.py		regression.py
requirements.txt		requirements.txt
retry.sh		retry.sh

License

Altinity/clickhouse-regression

Folders and files

Latest commit

History

Repository files navigation

🔬 Regression Tests for ClickHouse®

🗓 Scheduled Regression Runs

📅 Timetable of Scheduled Regression Runs

🔍 Table of Contents

Docker

Standalone docker-compose

x86_64

Aarch64 (ARM64)

Python modules

About

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Languages

Standalone `docker-compose`