Scoped coverage checking and confidence

[spacekitteh]

Testing my state machine code, I've noticed that when I have, say,

when (someRareCondition) do
    .... -- perform a pattern matching operation that only makes sense when someRareCondition is true
    cover 40 "pattern matches" (isJust sub)

then that seems to translate to being 40% of ALL cases, not only the cases when someRareCondition is true. Is there a way to take that into account? If I were to modify the predicate to not someRareCondition || isJust sub in the argument of cover, then the coverage numbers would be severely inflated by all the times that someRareCondition is false.

I had assumed that that was what the withConfidence functionality was for, but it seems to have no effect.

[ocharles]

I don't think it's possible to do what you want - how would Hedgehog know to only evaluate the coverage condition if when holds? I haven't thought about it deeply, but I wonder if logical implication is what you want?

cover 40 "pattern matches" (someRareCondition ==> isJust sub)

where

infixr 5 ==>
(==>) :: Bool -> Bool -> Bool
True  ==> x = x
False ==> _ = True

The other thing I'd suggest is to split your property test into two separate properties - one where someRareCondition in always True, and another where it is always False.

[spacekitteh]

but I wonder if logical implication is what you want?

That's precisely what this is: not someRareCondition || isJust sub. a ==> b === not a || b (at least classically!)

how would Hedgehog know to only evaluate the coverage condition if when holds?

If it's guarded by a when, then the call to cover will only ever occur when the when condition is true.

I haven't looked into it, but I suspect that the cover function increments a counter when the predicate is true, and after all the tests, simply divides the counter value by the number of tests performed. Could it not be tweaked to also increment a different count every time it's called, and instead of dividing by the total number of tests, just divide by the second counter?

The other thing I'd suggest is to split your property test into two separate properties - one where someRareCondition in always True, and another where it is always False.

This is in a Ensure Callback for a state machine test, and there are a lot of these different conditions+coverage checks I need to perform; it is infeasible to do this for all of them.

[spacekitteh]

I haven't looked into it, but I suspect that the cover function increments a counter when the predicate is true, and after all the tests, simply divides the counter value by the number of tests performed.

After checking the code, that does indeed seem to be the case. CoverCount could be modified to be a pair of numbers; instead of dividing by the number of tests in coverPercentage, it could just divide by the second number.

[ocharles]

That's precisely what this is

Ah yes, of course

After checking the code, that does indeed seem to be the case. CoverCount could be modified to be a pair of numbers; instead of dividing by the number of tests in coverPercentage, it could just divide by the second number.

Cool! I'm not very familiar with this part of Hedgehog, so you might want to ping one of the maintainers to confirm, but I'm sure a PR would be welcome!

[spacekitteh]

@jacobstanley is this correct?

[spacekitteh]

It'd probably be useful to have a cover and coverAbsolute, with the latter having the current functionality.

[ChickenProp]

Recommend against changing the behavior of the existing cover, but I do think it would be good to have the functionality you want. I guess the way to do it would be something like

• Make CoverCount a pair.
• toCoverCount turns NoCover into CoverCount 0 1 and Cover into CoverCount 1 1.
• Add a labelAbsolute :: Bool field to MkLabel. cover (and label, classify, collect) set this to true, some new function(s) set it to false.
• labelCovered takes its denominator to be either the test count (as currently) if labelAbsolute is true, or the second field of CoverCount if it's false.

Given #471 I think it would be worth thinking about the expected behavior if both existing and new functions get called with the same label.