Generate code coverage reports

[willfurnass]

In the first instance can generate interactively using pytest-cov and tox, then can move on to automating the measuring of code coverage using Travis CI and CodeCov.

[willfurnass]

I think it should be possible to generate code coverage using tox with:

pytest --maxfail=1 --cov={envsitepackagesdir}/mumot --nbval-lax --nbdime TestNotebooks/MuMoTtest.ipynb

However, that fails when we reach int1 = model1.integrate(...) with NoneType has no method integrate. Remove --cov={envsitepackagesdir}/mumot from the list of arguments to pytest and we don't encounter that error.

A theory: enabling code coverage changes the cell execution count numbering so the attempt to instantiate a model fails due to the issue described in #156.

[willfurnass]

Tried enabling code coverage report generation and uploading to codecov.io in PR #178 and 57f9928. The Travis CI build logs show code coverage report sent to stdout but attempts to upload report to codecov.io fail:

The command "tox" exited with 0.
0.27s$ bash <(curl -s https://codecov.io/bash)
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-0b37652
==> Travis CI detected.
    project root: .
    Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
==> Running gcov in . (disable via -X gcov)
==> Python coveragepy not found
==> Searching for coverage reports in:
    + .
--> No coverage report found.
    Please visit http://docs.codecov.io/docs/supported-languages
The command "bash <(curl -s https://codecov.io/bash)" exited with 0.
Done. Your build exited with 0.

Was the code coverage report written to disk by pytest-cov? Or was it written to a dir not checked by the codecov report uploader?

[jarmarshall]

A colleague has deployed this successfully on their GitHub repository I think - shall I ask for some further information?

[willfurnass]

@jarmarshall Thanks, might expedite things.

[neworderofjamie]

Hi @willfurnass - I am that college! Our current project is C++-based so we're using gcov to generate our coverage data meaning that things are a little different. However you should definitely be writing your coverage data to a file rather than stdout and, if the codecov script still isn't finding it, you can also manually point the script at your coverage data. For example we do this:
curl -s https://codecov.io/bash | bash -s - -f combined_coverage.txt -t OUR_MAGICAL_TOKEN
Where, as I remember, the -f forces codecov to look at a specific file.

[jarmarshall]

Thanks @neworderofjamie - does that help @willfurnass ?

[willfurnass]

Thanks @neworderofjamie; I'm getting a .coverage file generated when I run the test+coverage suite locally so I'm not sure why the codecov uploader isn't picking this up; I'll add a check to our travis config for the existence of .coverage at the end of a travis job. BTW it's good to see an example of uploading a file containing combined coverage data; that will likely be useful for this project in future for amalgamating data from unit and integration/regression tests.

[willfurnass]

Code coverage for the (stub of) a unit test suite (tests/test_all.py) now measured and pushed to codecov.io: https://codecov.io/gh/DiODeProject/MuMoT.

[willfurnass]

Can't generate code coverage with

pytest --cov={envsitepackagesdir}/mumot --maxfail=1 --nbval-lax --nbdime TestNotebooks/MuMoTtest.ipynb

as when pytest+nbval are run with coverage support enabled the explicit references to cell numbers are incorrect. Addressing #156 may help with this.

[willfurnass]

Coverage is still fairly low but I think the reporting is working now. Closing.

[jarmarshall]

@willfurnass if we can get code coverage working off notebooks as well (see issue #158 ?) this number should increase - in the short term, that would be preferred as we're short of time to write proper unit tests before publication...

[willfurnass]

At present we capture coverage data for (mostly stub) unit tests and from executing TestNotebooks/MuMoTtest.ipynb (without running regression tests on it); from tox.ini:

    pytest --cov={envsitepackagesdir}/mumot tests
    ...
    pytest --maxfail=1 --nbval-lax --nbdime --cov={envsitepackagesdir}/mumot --cov-append TestNotebooks/MuMoTtest.ipynb

Should we expand on this to capture coverage data when running through docs/MuMoTuserManual.ipynb too?

[willfurnass]

I think not; reopen if you disagree.