Detect empty test runs from additional runners by jay23606 · Pull Request #7 · dheerajjha/show-your-work · GitHub
Skip to content

Detect empty test runs from additional runners - #7

Merged
dheerajjha merged 2 commits into
dheerajjha:mainfrom
jay23606:pr-go/3-a-run-of-zero-tests-reads-as-passed-for-
Oct 20, 2026
Merged

dheerajjha merged 2 commits into
dheerajjha:mainfrom
jay23606:pr-go/3-a-run-of-zero-tests-reads-as-passed-for-

Conversation

@jay23606

Copy link
Copy Markdown
Contributor

Bug

Zero-test summaries from RSpec, ExUnit, Swift, Deno, and ctest were classified as passed.

Change

Recognize the reported empty-run signatures in the outcome detector while retaining the existing non-zero passed-count guard. Added one regression test covering all five empty signatures and non-empty examples.

Checks

  • node --test test/outcome.test.mjs → 10 tests passed.
  • node --test → 163 tests passed.

Regression test: fails on base twice (/'passed' !== 'empty'/), passes on fix twice (node --test test/outcome.test.mjs)

Fixes #3

@jay23606

Copy link
Copy Markdown
Contributor Author

@dheerajjha dheerajjha left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and contained. I ran 2c6a95d: 163 passed, and CI is green on all ten checks. On their own, every new signature reads as empty: a bare rspec, ExUnit, Swift or ctest summary, and deno's | 0 passed |, including an all-filtered run.

One problem before I merge, in the direction CONTRIBUTING worries about most. A run that executed tests can now read as empty, because some runners print one summary per suite or per app:

Test Suite 'EmptyTests' passed
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'MathTests' passed
	 Executed 4 tests, with 0 failures (0 unexpected) in 0.012 (0.013) seconds
Test Suite 'All tests' passed
	 Executed 4 tests, with 0 failures (0 unexpected) in 0.012 (0.015) seconds

That's swift test with one empty test class, and outcomeOf now calls it empty. An Elixir umbrella does the same thing: 0 tests, 0 failures from one app and 12 tests, 0 failures from another reads as empty too. Either way the Stop hook would stop an agent whose tests really ran.

The guard on line 129 only knows the N passed form of "something ran". Teach it these runners' forms:

!/\b[1-9]\d* (?:passed|passing)\b|\bExecuted [1-9]\d* tests?\b|\b[1-9]\d* (?:tests?|examples?), \d+ failures?\b/.test(text)

I checked that against six outputs:

  • reads as run: the Swift multi-suite output above, the umbrella, and 3 examples, 0 failures;
  • reads as empty: a truly empty Swift run, a truly empty mix test, and an empty rspec run.

With the Swift half of it in place, the suite still passes 163. Please add the multi-suite Swift output and the umbrella as passed cases in your test.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>

@dheerajjha dheerajjha left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's it. On 0cf2e59, 164 pass (main has 162), and CI is green on all ten checks.

The six outputs from my review land where they should:

  • the Swift multi-suite run, the umbrella and 3 examples, 0 failures read as passed;
  • an empty Swift run, an empty mix test and an empty rspec run read as empty.

I tried four more, and they all come out right. An umbrella where every app is empty, a Swift run filtered down to nothing, and 0 tests, 0 failures, 3 excluded all read as empty. A single Swift test reads as passed.

This changes the detector the README's benchmark numbers come from, so I checked one more thing. I replayed all 216 benchmark transcripts through main's detector and yours, at every point where the Stop hook judged a session, 222 in all. Not one verdict changes, so the published numbers stand. As a control, a detector that calls every run empty changes 39 of 54, so the replay does notice when something moves.

Merging. Thank you, @jay23606. A false empty stops an agent whose tests really ran, and this keeps those runners out of that trap.

@dheerajjha
dheerajjha merged commit 4d37d12 into dheerajjha:main Oct 20, 2026
10 checks passed
dheerajjha added a commit that referenced this pull request Oct 20, 2026
@jay23606
jay23606 deleted the pr-go/3-a-run-of-zero-tests-reads-as-passed-for- branch October 21, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A run of zero tests reads as passed for rspec, ExUnit, swift, deno and ctest

2 participants