Scenario execution priority attributes

[50U10FCA7]

Priority attribute tells the executor how early need to add scenario to the execution queue.

In some cases there is no way to tell the executor to run scenarios in specific order.
For example: Scenarios that will be executing for a long time must be added to execution queue before others.

Design proposal:

@priority [integer]
Scenario: my slow scenario

Scenarios with lower priority value will be added to execution queue earlier (or vice versa).

[ilslv]

@50U10FCA7 @tyranron

For example: Scenarios that will be executing for a long time must be added to execution queue before others.

To really solve this issue, in addition to the @priority=n tag, we improve default Runner. For now, in case of concurrent steps, it collects batch of the Steps to execute and waits for until all are complete. We should start executing next Step right after any of them is finished.
Also maybe we should leave an option to wait for all Steps in the batch, as it may help with distributing the load on tested application.

[tyranron]

@ilslv I'm not sure this is the responsibility of a Runner. It seems that a Parser may decide which Features should be fed first to the Runner (however, Runner still should be able to distinguish that on Scenario level).

This priority issue doesn't have high priority for us at the moment 😊
We can give it some time and think about better designs.

We should start executing next Step right after any of them is finished.

That would be worthful improvement anyway.

Also maybe we should leave an option to wait for all Steps in the batch, as it may help with distributing the load on tested application.

How it should help? I don't really grasp your idea.

[ilslv]

@tyranron

How it should help? I don't really grasp your idea.

If in the batch there is Scenario which is very heavy to execute and other concurrent Scenarios may throttle it, our batch approach may be helpful, as batch will shrink over time and lower load on the tested target. But maybe it's better to use @serial in such cases.

[tyranron]

@ilslv yup, or even lower/higher @priority.

[Solonotix]

Note: I'm an outsider that's new to the Rust community. I just stumbled on this project because I wanted to potentially explore Rust-Cucumber as an option.

Wouldn't priority of scenario be handled by the default parsing behavior of Gherkin files? Maybe this project is different, but the default behavior is to sort steps by the feature file name (maybe file path), then the top-to-bottom order of scenarios. If you want to enforce a priority order, preface the features with a numeral that represents your desired order, and you can even use a dense-ranking system for scenarios that don't need to be run before or after a particular group.

Example:

\---features
        01-runs-first.feature
        02-runs-next.feature
        03-indeterminate-test.feature
        03-still-dont-care.feature
        03-technically-fifth.feature
        99-must-run-last.feature

[tyranron]

@Solonotix this library works the following way:

1. The whole execution pipeline consists of Parser -> Runner -> Writers.
2. The parser of Gherkin files is abstracted in the way to be just a Stream of parsed Features. It could be anything emitting Features, not just glob-walker over fs .feature files.
3. Runner polls the Parser's Stream and executes Scenarios concurrently (the concurrency level can be configured).
4. Writer receives events about execution status from Runner in out-of-order manner, but the writer::Normalized normalizes the output in the way that once some Feature is finished, it's fully printed right away.

So, at the moment, we're not preserving any order when executing and printing Scenarios, only steps inside a Scenario are executed sequentially.

Such concurrent execution allows us to run ~1000 E2E scenarios on a GraphQL API endpoint in nearly 10-15 minutes. Some of those scenarios test expiration mechanisms and wait for 3-5 minutes for the expiration to happen. Some of them are @serial (isolated) as require restarting GraphQL API instance (testing failure recovering).
Before introducing concurrent execution, our test suite was about 400 scenarios only, and ran for more than an hour.

[ilslv]

@Solonotix I also would argue that Scenarios should be distributed across .feature-files based on some domain-specific needs, not execution priority.