Test Execution behaves differently from vstest ado task vs vstest.console.exe

[hankweber]

Description

We are observing differences in test exectuion behavior when executing with the vstest ado task versus just invoking vstest.console.exe manually. What we are seeing is that each batch of test cases is run in it's own test session when running via the ado task whereas all test cases are run in the same test session when running manually with vstest.console.exe.

Steps to reproduce

Run a pipeline (example here) that uses the vstest task and test cases will be run in their own test session. Run the equivalent runsettings/configuration manually with vstest.console.exe and all test cases will be run in the same test session.

Expected behavior

The test case/session behavior should be the same regardless invocation.

Actual behavior

The test case/session behavior is different between the vstest task and the vstest.console.exe

Diagnostic logs

Slice from vstest.console.exe.log
Slice1-from-vstest-ado-task.log
Slice2-from-vstest-ado-task.log

Environment

[nohwnd]

ADO task is automating vstest.console to provide additional functionality. If you need a way to run all test cases in one sessions then that issues should be raised in https://github.com/microsoft/azure-pipelines-tasks.

[hankweber]

ok, I think we are looking to have the same behavior outside the ado task. so we want to execute our tests with each test case in its own session. I'll poke around the pipeline task code and see if I can see where they're doing that so we can copy it

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform