We should use test.fixme() to mark tests that are fixable but currently failing and use test.skip() for those which are not expected to pass in Bidi mode ever. We can then look at the annotations and if it contains fixme then the test didn't run because it is known to timeout.

This is great. Thanks for getting the CSV reporter updated that quickly!

Now we have closer numbers for the list of failures but there is still a discrepancy of 73 between the results as reported in CI which are 3664 (all) - 2718 (passing) = 958 to the 885 reported entries in the CVS. Do you have an idea what these could be?

As far as I can see, for this run the stats reported in the terminal:

3676 tests using 2 workers

731 failed
227 skipped
2718 passed

These add up just fine.

Then in the .csv report there are 907 lines with the failure results which include both actual failures of the tests that were executed and failed (731) as well as some tests that we marked as fixme based on the expectations file. Maybe skipping tests based on the expectations files does more disservice by confusing people than saving resources, we can remove them altogether if you like. Alternatively, we can export all test results to the report.csv rather than failures and fixme (here is the PR: #34199). That way the total in the terminal and in the .csv report will always match. WDYT?

Then in the .csv report there are 907 lines with the failure results which include both actual failures of the tests that were executed and failed (731) as well as some tests that we marked as fixme based on the expectations file. Maybe skipping tests based on the expectations files does more disservice by confusing people than saving resources, we can remove them altogether if you like.

Ah, there it is coming from! For now I would be fine given that we understand where the gap is coming from. Not sure what you mean with remove them altogether. Are you saying those should still be run even when they timeout and the tests take longer to run? I would still like to see all the tests running and maybe we take a look at getting an understanding of why tests are timing out first and get those fixed if easily possible (like the page.url() issue that you fixed). Once we have the easy ones fixed we could enable all tests?

Alternatively, we can export all test results to the report.csv rather than failures and fixme (here is the PR: #34199). That way the total in the terminal and in the .csv report will always match. WDYT?

The problem here is that we need special filtering in the Google spreadsheet so that we ignore or immediately remove all the passing tests when we import the data. I assume that we would have to write an App Script to do that instead of just having the =IMPORTDATA function running. I wont have the time for that right now.