Skip to content

Commit

Permalink
Merge bitcoin/bitcoin#31489: fuzz: Fix test_runner error reporting
Browse files Browse the repository at this point in the history
fa0e30b fuzz: Fix test_runner error reporting (MarcoFalke)

Pull request description:

  The error reporting is confusing, because right now it prints:

  https://cirrus-ci.com/task/4846031060336640?logs=ci#L4931

  ```
  ...
  Traceback (most recent call last):
    File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 411, in <module>
      main()
    File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 199, in main
      run_once(
    File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 376, in run_once
      assert len(done_stat) == 1
             ^^^^^^^^^^^^^^^^^^^
  AssertionError
  ```

  This is harmless, but confusing.

  Fix it by collecting statistics only when the program has not aborted. (Can be reviewed with `--color-moved=dimmed-zebra`)

  Also, reword the error message to align it with error messages in other test_runners in this repo.

ACKs for top commit:
  dergoegge:
    utACK fa0e30b
  brunoerg:
    code review ACK fa0e30b
  marcofleon:
    Tested ACK fa0e30b. Prints out the error for the target that crashed. Much clearer than the current error message.

Tree-SHA512: 5e8d3fc0e4837b3264ff0c3cb322fe7fe2ec7af48d35e2a14f82080d03ace793963c3314611b0a170a38e200497d7ba703d9c35c9a7ed3272d93e43f0f0e4c2b
  • Loading branch information
fanquake committed Dec 17, 2024
2 parents 1251a23 + fa0e30b commit 785486a
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions test/fuzz/test_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -371,19 +371,19 @@ def job(t, args):
for future in as_completed(jobs):
output, result, target = future.result()
logging.debug(output)
if using_libfuzzer:
done_stat = [l for l in output.splitlines() if "DONE" in l]
assert len(done_stat) == 1
stats.append((target, done_stat[0]))
try:
result.check_returncode()
except subprocess.CalledProcessError as e:
if e.stdout:
logging.info(e.stdout)
if e.stderr:
logging.info(e.stderr)
logging.info(f"Target {result.args} failed with exit code {e.returncode}")
logging.info(f"⚠️ Failure generated from target with exit code {e.returncode}: {result.args}")
sys.exit(1)
if using_libfuzzer:
done_stat = [l for l in output.splitlines() if "DONE" in l]
assert len(done_stat) == 1
stats.append((target, done_stat[0]))

if using_libfuzzer:
print("Summary:")
Expand Down

0 comments on commit 785486a

Please sign in to comment.