Measure code coverage

[IlanCosman]

Fishtape is a great testing framework, but something that is missing is a description of how much code your tests actually cover.

Using fish --debug-output and fish_trace it's easy to dump the code that was run into a file and then count the lines.

Then, using a tool like cloc one can get the number of fish code lines in the user's project to calculate the percent code coverage.

Steps?

1. I'm not sure how fishtape's subshells work, but presumably they could add fish --debug-output and fish_trace
2. A lot of code is repeated when running a test file, so duplicate code sections must be removed
3. Fishtape has it's own code that would need to be silenced/removed from the output
4. Somehow get only the number of code lines from cloc. Probably have to go through yaml or something, I don't think there's a way of just printing one number.

This is somewhat outside the scope of the project, so if there's no interest in implementing this that's totally fine 😄 I know it would be useful to myself and likely others though.

[jorgebucaran]

@IlanCosman Hey! Late reply here. I agree this would be amazing. I'm not 100% sure it belongs to fishtape or not, but we can think about that later. Maybe something as simple as (notice the last two lines):

fishtape --coverage ~/bax.fish/test/*.fish 
TAP version 13
# bax.fish
ok 1 stdout
not ok 2 export
  ---
    operator: 
    expected: a valid operator
    actual:   = _bax=foo
  ...
ok 3 unset
ok 4 $PATH
ok 5 alias
ok 6 cd
ok 7 exit status
not ok 8 semi
  ---
    operator: 
    expected: a valid operator
    actual:   = _bax=semi
  ...
ok 9 dollar

1..9
# pass 7
# fail 2
# bax.fish 98/100 
# ok 70%

The documentation on how to use --debug-output and fish_trace is not the "best", but I guess we could figure it out. Is this something you want to give a shot yourself or just would like to have?

[IlanCosman]

Thanks for the reply Jorge. I really like that mockup! 😄I'm not sure whether it belongs in fishtape either, but I don't really see where else it would go since there aren't any other fish testing suites.

To be entirely honest and maybe a bit selfish 😔, at the moment I'd just "like to have" it. The startup cost for me is a bit high in proportion to it's priority level, involving learning awk and then understanding your code. If you're not interested I might try it out at some point, though likely not soon since I'd like to focus on making Tide async.