[Feature] Don't require recommissioning the DUT for every test

[cecille]

Feature description

With the time required to factory reset the device, delay starting the commissioning, delay starting the test, and re-running the failed test, this add hours to the testing process even for small devices.

Use Cases

Commission device once, run all tests.

Test Cases

just run through multiple tests in a row

Additional Information

No response

[fabiowmm]

@cecille The current logic in TH for running python tests is as follows:

• Python Testing Suite: these are test scripts for which the script author wants TH to perform commissioning before launching the first script in the run (and only that)
• Python Testing Suite - No commissioning: These are the ones for which TH will not try to commission (if commissioning is needed it will either be done manually by the user or done by the script itself).
• Python Testing Suite - Old script format: For these ones TH will ask the user, before launching EVERY script in the run, whether it should try to commission the device before launching the script.

The inclusion of test scripts in each category above is determined by the value of the is_commissioning variable defined in the script code (True means type 1, False means type 2 and undefined/none means type 3).

So, except for the Old script format category, TH does not try to commission before each test.

[cecille]

This is a slightly different problem - I think I mean re-commissioning at each run. The TH often fails to start tests, resulting in test runs with partial failures. To re-run these tests, you need to create a new test run with only the failing tests etc. until all the tests are run. The back and forth of creating new test runs with only the failing tests is challenging, as is identifying which run holds the pass result for a given test. As a result the ATLs seem to have developed some workarounds. One such workaround is their recommendation that if you're going to use the UI that test be run one at a time, one per run, so you can just re-run the entire run and see the passing result.

This makes it easier to determine when the correct set of tests has been run, but it also means that the device needs to be recommissioned before every time. The first thing the TH seems to do before each test run is to ask you to put the device in commissioning mode.

Storing the admin kvs between runs would allow folks to run multiple test runs without re-commissioning.