-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Should size grow faster when the number of tests is small? #472
Comments
This was referenced Jan 30, 2023
ChickenProp
added a commit
to ChickenProp/haskell-hedgehog
that referenced
this issue
Jan 30, 2023
Closes hedgehogqa#472. We now grow test size in fixed increments from 0 to 99, or as close as we can without going over, in however many tests we run. If we run more than n * 100 tests, then we go from 0 to 99 n times, and then do fixed increments for the remainder. Additionally, if we discard a bunch of times in a row we start to grow the size.
This was referenced Jan 30, 2023
ChickenProp
added a commit
to ChickenProp/haskell-hedgehog
that referenced
this issue
Mar 14, 2024
Closes hedgehogqa#472. We now grow test size in fixed increments from 0 to 99, or as close as we can without going over, in however many tests we run. If we run more than n * 100 tests, then we go from 0 to 99 n times, and then do fixed increments for the remainder. Additionally, if we discard a bunch of times in a row we start to grow the size.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I often don't want to run a full set of tests and use
NoConfidenceTermination 10
. (Specifically I usemodifyMaxSuccess (const 10)
from hspec-hedgehog). But that means the size never grows above 9, soRange.linear 0 10
will always generate 0, which isn't what I expected. It works out okay if I write my generators expecting a specific number of max-tests, or I can callGen.scale
everywhere. But it would be more convenient if the size would grow to something large in a small number of tests.Quickcheck does something like this. I don't fully understand the arithmetic but in this case the size would go 0, 10, 20, ..., 90. (
n
is the test number andd
is a discard count.)I don't think it has an analogue of
EarlyTermination
orNoEarlyTermination
, so hedgehog would need to figure out what to do about that. I don't use these myself, but from what I gather from the code:NoEarlyTermination c t
still runs exactlyt
tests, the same asNoConfidenceTermination t
, so it could have the same behavior asNoConfidenceTermination
.EarlyTermination c t
actually ignorest
and runs at least 100 tests, more if needed for the confidence thresholdc
, but always some multiple of 100. (Is this intended? It's not what the name suggests to me.) So it could just keep the existing behavior.So these don't seem like big problems to me.
The text was updated successfully, but these errors were encountered: