Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-run failures only #43

Open
gap777 opened this issue Nov 20, 2023 · 6 comments
Open

Re-run failures only #43

gap777 opened this issue Nov 20, 2023 · 6 comments
Labels
enhancement New feature or request

Comments

@gap777
Copy link

gap777 commented Nov 20, 2023

Is it possible to re-run the tests limited to those that failed on the last execution?

@ilyazub
Copy link
Collaborator

ilyazub commented Nov 21, 2023

@gap777 I use rspec --only-failures for that. Currently, this is not supported with turbo_tests.

I expect that passing the extra options to RSpec will be enough to support this feature (#14 (comment)).

@ilyazub ilyazub added the enhancement New feature or request label Nov 21, 2023
@douglasshuang
Copy link

I work with @gap777.

I realized that rspec --only-failures couldn't find failed examples reliably because the parallel runners invoked by turbo_tests were writing to the same status file and clobbering each other's output. I changed our spec/spec_helper.rb to use a different file name per runner:

  config.example_status_persistence_file_path = "spec/examples#{ENV['TEST_ENV_NUMBER']}.txt"

We already use a shell script to invoke turbo_tests, so I added a liine to consolidate those files at the end:

cat spec/examples?*.txt > spec/examples.txt

So for now, we can at least run (non-parallel) rspec --only-failures after turbo_tests.

@ilyazub
Copy link
Collaborator

ilyazub commented Nov 22, 2023

@douglasshuang Thanks to your feedback!

Please note that combining multiple examples*.txt may lead to the RSpec failure. RSpec status persistence files have a header which is expected to be present once (ref: ExampleStatusParser#initialize).

# Concatenate multiple files with the same header
# https://unix.stackexchange.com/a/170692/33308
$ (head -2 spec/examples1.txt && tail -n +3 -q spec/examples?*.txt) > spec/examples.txt

# Checking it worked
$ wc -l spec/examples*.txt
   73015 spec/examples.txt
   46467 spec/examples1.txt
   26550 spec/examples2.txt
  146032 total

# Header is present only once
$ grep example_id spec/examples*.txt
spec/examples.txt:example_id                                                                                                 | status  | run_time               |
spec/examples1.txt:example_id                                                                                                 | status  | run_time               |
spec/examples2.txt:example_id                                                                                                 | status  | run_time               |

@douglasshuang
Copy link

@ilyazub Thank you very much for the refinement!

@santigolucass
Copy link

@douglasshuang Hey, what was your final solution to run the failed specs in nonparallel?

I would like to talk to you about your parallel process suit. We can do it via email, discord, or whatever you find best

Really appreciate, thanks

@douglasshuang
Copy link

@santigolucass Hi, here is the complete solution:

First, we configure RSpec to use a different status persistence file per test runner.

spec/spec_helper.rb:

RSpec.configure do |config|
  ...
  config.example_status_persistence_file_path = "spec/examples#{ENV['TEST_ENV_NUMBER']}.txt"
  ...
end

This produces files named as follows when running turbo_tests:

  • spec/examples1.txt
  • spec/examples2.txt
  • ...

Then, after we run turbo_tests, we combine the files, keeping only one header line.

bin/turbo_tests:

#!/bin/bash

bundle exec turbo_tests $*
(head -n 2 spec/examples1.txt && tail -n +3 -q spec/examples?*.txt) > spec/examples.txt

Since ENV['TEST_ENV_NUMBER'] is not set when running rspec normally, it will use spec/examples.txt:

$ bundle exec rspec --only-failures # This should work as expected now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants