-
Notifications
You must be signed in to change notification settings - Fork 346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate flaky com.twitter.algebird.CollectionSpecification
test
#856
Comments
I'm looking for a good first issue to look at in this repo, and am not able to reproduce:
Under what conditions does this test fail? |
some of the tests in this repo are statistical tests about probability data structures. When you make some claim, it can often be false with some small probability. When we run the tests over and over (CI for different versions, or dependency updates) we tend to hit these. If we weaken the tests so much that never happens, then the tests are also very weak (in some cases trivial, in that they would always pass). Another alternative is to fix a seed so we don't reseed with the time on these tests. |
Maybe @regadas can post a link to a build that failed and triggered this issue, but flaky here means that the failure is random and intermittent due to generative testing. So the task would be to find a build around when this issue was filed that failed this test, and then run a unit test version of the generative test with the generated values that caused the failure. That will help you track down the bug. |
Jinx @johnynek ! |
Got it. I'll look into fixing a seed: that sounds like a reasonably clean solution to taking care of all these flaky tests if the maintainers are fine without a randomized seed. |
Hi, y'all! I believe I did a bad job here since I didn't attach the relevant info and now the build logs are gone so pinpointing this will be hard. Sorry about that! 🙏 Locally IIRC, I wasn't able to replicate it. So... I guess we can wait and see if some build fails again with it and update this accordingly? 🤷 |
Haven't looked into it yet but got a build failure related to this. Adding this to the issue body as well. https://github.com/twitter/algebird/runs/7812911450?check_suite_focus=true
|
Good example, and I bet manually reducing the dimension way down would still trigger a failure. A good case! |
https://github.com/twitter/algebird/runs/7812911450?check_suite_focus=true
The text was updated successfully, but these errors were encountered: