[Question] "On Retry" is > 0. But where are we retrying those tests?

[jonathanyeong]

Hey team, we migrated to using the new bktec (from v0.8.0!) and enjoying the extra observability so far. I had a question around "On retry". I see on retry greater then 0, but I don't see bktec actually retrying the test. Am I understanding this value correctly?

If we compare to an actual failing test, I can see that bktec shows the retries

For context, migrating from 0.8.0 has resulted in test times that are overall a minute or two slower. So I wonder if that's the retry mechanism doing its thing.

[nprizal]

Hey @jonathanyeong, thanks for reaching out. We’re facing a similar issue in Buildkite. It turns out that there are a few tests that share the same description. bktec and Test Engine use the combination of scope (example group description in Rspec) and name (example description in Rspec) to identify tests. If there are tests with the same scope and name, bktec reports them as passed on retry because it mistakenly believes that the “same” test has already been run.

We’re investigating this issue and will see if we can find a solution. In the meantime, could you please check if all test descriptions are unique? If you’re using RSpec (and Rubocop), you can use RSpec/RepeatedDescription and RSpec/RepeatedExampleGroupDescription to check for duplicates.

As for the test getting slower, there could be several factors contributing to this. Retry is one of them. I wouldn’t expect upgrading from 0.8.0 to 1.2.0 to slow down your test suites. To confirm if the test suite is indeed getting slower, you can check your test runs from the Test Splitting page in your Buildkite Test Suite. You can also look into individual runs to see the impact of job retry.