Numpy getting included in coverage numbers when imported

[sg3-141-592]

Where numpy is imported into a fuzz.py file it's getting included in the reachability and coverage numbers

A couple of examples

• bottle - introspector report - 2023-05-21
• toml - introspector report - 2023-05-21

For toml I tried setting export PYFUZZPACKAGE=$SRC/toml/toml which fixed just analysing the toml functions but breaks the reachability analysis.

[DavidKorczynski]

I think this is likely something we cannot resolve due to the way coverage is collected.

Probably, when import numpy is executed then various functions in the numpy module tree will be called, meaning numpy code other than def declarations will be called triggering the actual logic of the functions.

One way we could solve this is by making it such that code coverage collection begins only right before fuzzing launches, rather than, e.g. before all imports. However, this won't be perfect as well since we can still have imports triggered by the target code, although this will be done much more rarely.

Another option is to collect code coverage of the fuzzer but without any actual fuzzing and then subtract the code coverage found with a run of code coverage where the fuzzing is actually run. This way we could remove any code coverage gained from non-fuzzing code.

Another option is to make it possible for projects to add environment variables specifying which paths should be included in the code coverage reports.