Wait replication-sync returns valid exit-code while CH connection error

[MedvedewEM]

In case of stopped CH list_table_replicas(ctx) query returns also ConnectionErrors.
Let's except these exceptions once outside for-loop.

[MikhailBurdukov]

Why we should exit here? Is reraise the exception + logging.error or raise new exception with custom msg not enough?

Also maybe for requests.exceptions.ConnectionError add retries? Because we can have some short network issues

[MedvedewEM]

Why we should exit here? Is reraise the exception + logging.error or raise new exception with custom msg not enough?

That is good point - what do you think about fix problem globally here - https://github.com/yandex/ch-tools/blob/main/ch_tools/chadmin/cli/chadmin_group.py#L54
Just adding raise in except branch. I afraid it is initial problem. And not only for wait replication-sync.

Also maybe for requests.exceptions.ConnectionError add retries? Because we can have some short network issues

It is already here - https://github.com/yandex/ch-tools/blob/main/ch_tools/common/clickhouse/client/clickhouse_client.py#L165

[MikhailBurdukov]

Yeah, we need to review all such emergency exists, we got a lot of them:

./ch_tools/s3_credentials/main.py:        sys.exit(1)
./ch_tools/s3_credentials/main.py:        sys.exit(1)
./ch_tools/s3_credentials/main.py:    sys.exit(0)
./ch_tools/monrun_checks/main.py:            sys.exit(1)
./ch_tools/monrun_checks/main.py:                sys.exit(1)
./ch_tools/chadmin/cli/wait_group.py:            sys.exit(1)
./ch_tools/chadmin/cli/wait_group.py:                sys.exit(1)
./ch_tools/chadmin/cli/wait_group.py:            sys.exit(0)
./ch_tools/chadmin/cli/wait_group.py:    sys.exit(1)
./ch_tools/chadmin/cli/wait_group.py:        sys.exit(0)
./ch_tools/chadmin/cli/wait_group.py:    sys.exit(1)
./ch_tools/chadmin/cli/zookeeper_group.py:    sys.exit(1)

Let's deal with it in the separate pr

It is already here

Thanks, missed it.

[MikhailBurdukov]

@MedvedewEM could you look into? https://github.com/yandex/ch-tools/actions/runs/10849461150/job/30108748612
Seems related

[MedvedewEM]

@MedvedewEM could you look into? https://github.com/yandex/ch-tools/actions/runs/10849461150/job/30108748612 Seems related

It is weird. Second retry was helpful.
It could be flap, but right now I do not have a clue why stopping fetches on node1 and calling chadmin wait replication-lag on the same node1 could return different timeout's exceptions (http.ReadTimeout vs ClickHouse.Timeout).
It should be same exception for each run...

Lets watch for this potential flap in coming PRs.