Repository navigation
Detect empty test runs from additional runners - #7
dheerajjha merged 2 commits into
Conversation
dheerajjha
left a comment
There was a problem hiding this comment.
Nice and contained. I ran 2c6a95d: 163 passed, and CI is green on all ten checks. On their own, every new signature reads as empty: a bare rspec, ExUnit, Swift or ctest summary, and deno's | 0 passed |, including an all-filtered run.
One problem before I merge, in the direction CONTRIBUTING worries about most. A run that executed tests can now read as empty, because some runners print one summary per suite or per app:
Test Suite 'EmptyTests' passed
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'MathTests' passed
Executed 4 tests, with 0 failures (0 unexpected) in 0.012 (0.013) seconds
Test Suite 'All tests' passed
Executed 4 tests, with 0 failures (0 unexpected) in 0.012 (0.015) seconds
That's swift test with one empty test class, and outcomeOf now calls it empty. An Elixir umbrella does the same thing: 0 tests, 0 failures from one app and 12 tests, 0 failures from another reads as empty too. Either way the Stop hook would stop an agent whose tests really ran.
The guard on line 129 only knows the N passed form of "something ran". Teach it these runners' forms:
!/\b[1-9]\d* (?:passed|passing)\b|\bExecuted [1-9]\d* tests?\b|\b[1-9]\d* (?:tests?|examples?), \d+ failures?\b/.test(text)I checked that against six outputs:
- reads as run: the Swift multi-suite output above, the umbrella, and
3 examples, 0 failures; - reads as empty: a truly empty Swift run, a truly empty
mix test, and an empty rspec run.
With the Swift half of it in place, the suite still passes 163. Please add the multi-suite Swift output and the umbrella as passed cases in your test.
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>
dheerajjha
left a comment
There was a problem hiding this comment.
That's it. On 0cf2e59, 164 pass (main has 162), and CI is green on all ten checks.
The six outputs from my review land where they should:
- the Swift multi-suite run, the umbrella and
3 examples, 0 failuresread as passed; - an empty Swift run, an empty
mix testand an empty rspec run read as empty.
I tried four more, and they all come out right. An umbrella where every app is empty, a Swift run filtered down to nothing, and 0 tests, 0 failures, 3 excluded all read as empty. A single Swift test reads as passed.
This changes the detector the README's benchmark numbers come from, so I checked one more thing. I replayed all 216 benchmark transcripts through main's detector and yours, at every point where the Stop hook judged a session, 222 in all. Not one verdict changes, so the published numbers stand. As a control, a detector that calls every run empty changes 39 of 54, so the replay does notice when something moves.
Merging. Thank you, @jay23606. A false empty stops an agent whose tests really ran, and this keeps those runners out of that trap.

Bug
Zero-test summaries from RSpec, ExUnit, Swift, Deno, and ctest were classified as passed.
Change
Recognize the reported empty-run signatures in the outcome detector while retaining the existing non-zero passed-count guard. Added one regression test covering all five empty signatures and non-empty examples.
Checks
node --test test/outcome.test.mjs→ 10 tests passed.node --test→ 163 tests passed.Regression test: fails on base twice (/'passed' !== 'empty'/), passes on fix twice (
node --test test/outcome.test.mjs)Fixes #3