Test suite running in parallel is undersupported

[Corion]

Enabling test suite parallelism for installations from CPAN is problematic.

There is only the option of running all tests of a distribution in parallel, without any way for the distribution itself to influence that, by setting HARNESS_OPTIONS ( Test::Harness ) or creating a ~/.proverc file ( App::Prove ). This mechanism is currently used for two purposes:

1. Specifying a maximum limit for the test suite parallelism
2. Enabling test suite parallelism for the test suite

This is unfortunate, because not all test suites are safe for running in parallel and there is currently no way for specifying within a distribution how its test suite can be run in parallel.

A distribution should be able to opt-in into running (parts of) its test suite in parallel, instead of being forced to be run in parallel.

Currently, Test::Harness (together with App::Prove and make test) has only the most basic support for running the test suite of a distribution in parallel. Supplying additional information about which tests can be run in parallel is possible for App::prove ( but maybe see #75 ) on the command line and for Test::Harness by presence of a testrules.yml file. App::prove could respect a testrules.yml file with some small changes.

Ideally, specifying the maximum level of parallelism does not automatically enable full/blind test suite parallelism in Test::Harness and App::Prove. A solution to allow distributions and the local user to configure test suite parallelism could be the presence or absence of a testrules.yml file:

a) If a testrules.yml is present, this should be respected together with the -j option from HARNESS_OPTIONS
b) If HARNESS_OPTIONS contains (say) --parallel-default=~/.testrules.yml , this is used in the case where no testrules.yml exists in the distribution
c) -j8 does not enable test suite parallelism on its own anymore

Of course, c) means breaking behaviour, which may or may not be that great. It can be worked around by implementing something like b) .

[Leont]

App::prove could respect a testrules.yml file with some small changes.

If that doesn't work than that's certainly not intentional, and should be made to work.

c) -j8 does not enable test suite parallelism on its own anymore

I don't think that's a desirable default.

[Corion]

Does test suite parallelism usually work out?

At least the test suites for WWW::Mechanize::Firefox and WWW::Mechanize::PhantomJS do not work well when run in parallel. I'm just releasing fixes for the test suite and the main module WWW::Mechanize::Chrome so that these will actually work in parallel.

Basically all test suites that do some TCP connections and scan for a free port will likely have problems when run in parallel.

[Corion]

I retract my wrong assertion that App::prove does not respect a testrules.yml. It does.

The rest of this issue relies on whether "we" prefer to enable parallelism and expect non-parallel test suites to add an appropriate testrules.yml or not.

[karenetheridge]

My opinion is that cpan modules should support running tests in parallel. Usually failures are a simple matter such as a hardcoded filename for a temp file and are easily fixed. If there is truly an issue, problematic tests can be put in one file to guarantee they are run serially.

[andk]

TCP connections and scan for a free port
Did you try Test::TCP and Proc::Guard? Do they not solve all TCP tasks?

Does test suite parallelism usually work out?
My impression is yes.

the -j option
I think the problem is that determining a good value for -j is not trivial for the author of a test and the owner of a smoker. Much depends on what other parties are doing on a system. Best bet is to program defensively. Any resource may be limited at any time, don't they? And if I run two smokers in parallel it may happen that two different perls run the same test suite at the same time.