Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Surprises wrt --state option vs. file arguments #35

Open
andk opened this issue Feb 21, 2014 · 3 comments
Open

Surprises wrt --state option vs. file arguments #35

andk opened this issue Feb 21, 2014 · 3 comments

Comments

@andk
Copy link
Member

andk commented Feb 21, 2014

What --state does when combined with file arguments is underdocumented
and sometimes surprising. The following in the Test-Harness
distribution directory:

Options + Arguments what is does ok/surprise
rm .prove
--state=slow,save refuses to test surprise[0]
rm .prove
--state=save runs all tests ok
--state=save,fast t/parse.t runs all tests surprise[1]
rm .prove
--state=save,fast t/parse.t runs parse.t surprise[2]
--state=save,fast t/base.t runs parse.t and base.t
--state=save,all,fast t/base.t runs parse.t and base.t surprise[3]

[0] because there are tests on disk and they would be run with
state=save alone; this is one of the cases where it would be helpful
to just run all tests from disk.

[1] what happens is that the argument t/parse.t is merged with all
files in .prove. Apparently there is no way to run a subset of the
known tests with state options without editing the .prove file.

[2] Given that surprise [0] above refused to work, it's now
surprising that this incantation does the right thing

[3] keyword "all" apparently does not mean all tests on disk. Not
mentioned in the docs.

I believe the currently undocumented rules are:

(1) as long as no statefile exists, any combination of state arguments
turn prove into a noop unless there are explicit file or directory
arguments on the commandline. E.g.

prove --state=save,all         # noop
prove --state=save,all t/1.t   # runs 1.t

(2) if a statefile exists, the set of tests to be considered to be run
is a union of the test files in the state file and files or
directories on the command line. From this union set all tests on
the command line are run unconditionally while files from the
statefile are run according to the rule. E.g.

rm -f .prove
prove --state=save        t/1.t t/2.t # runs 1.t and 2.t
prove --state=save,failed t/1.t       # runs 1.t and whatever failed above

(3) the keyword "all" refers to the files in the statefile, not to the
files on disk

I'm unsure whether one would prefer the rules to change or get these
rules documented. I find the rules hard to understand and to document
and also disputable.

@KES777
Copy link

KES777 commented Oct 5, 2018

Faced into problem [1] too.

[1] what happens is that the argument t/parse.t is merged with all
files in .prove. Apparently there is no way to run a subset of the
known tests with state options without editing the .prove file.

To my mind that was bad idea to allow --state multiple times.

There is already ability --state=hot,save

There is not clear how these two will be merged: --state=hot,save --state=slow,failed

@KES777
Copy link

KES777 commented Oct 5, 2018

also [2]

prove --state=save t/OpenAPI3

prove --state=failed t/OpenAPI3

Here I expect that only failed tests from t/OpenAPI3 will be running

To my mind provided list of tests must be filtered by --state

@KES777
Copy link

KES777 commented Oct 5, 2018

Also this belongs to --state option:

Tests that have never failed will not be selected. To run all tests with the most recently failed first use
$ prove -b --state=hot,all,save

kes@work ~/t $ prove --state=hot,all,save
t/OpenAPI3/pointer.t ................... ok                             
t/OpenAPI3/types.t ..................... ok                             
t/OpenAPI3/keywords.t .................. ok                             
t/OpenAPI3/empty.t ..................... ok                             
===(      32;1  6/?  0/?  0/?  6/6  0/?  0/? )==========================
#   Failed test '404 Not Found'
#   at t/OpenAPI3/exception.t line 29.
#          got: '501'
#     expected: '404'

#   Failed test 'exact match for JSON Pointer "/code"'
#   at t/OpenAPI3/exception.t line 29.
#          got: undef

As you can see failed test are not run first.

Next also has not effect:

$prove --state=hot,slow,save

Here I expect to run most recently failed first, then slowest, then save test result

This occur when ~/.proverc file exists and there is already have --state option

Even if the content of .proverc will be just:

--state=hot,all,save
--jobs=9
-l

and I do:

$ prove
t/OpenAPI3/types.t ..................... ok                             
t/OpenAPI3/keywords.t .................. ok                             
t/OpenAPI3/pointer.t ................... ok                             
===(       4;0  1/?  0/?  0/?  0/?  0/?  0/?  0/? )=====================
#   Failed test '404 Not Found'
#   at t/OpenAPI3/exception.t line 29.
#          got: '501'
#     expected: '404'

as you can see testing do not run hot tests first

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants