{{ message }}
Repository navigation
Stop asserting simulator reliability in testResultStatusCount - #397
Merged
Merged
Conversation
The Codecov run on main failed while the Test run on the same commit passed: 53adfaf produced Passed(6)/Failed(6) instead of Passed(7)/Failed(5). FirstSuite.testOne appears as both passed and failed across the fixture generation logs. Its body is XCTAssert(true) plus an attachment, so it cannot fail on its own assertion — the failure comes from setUp, which calls XCUIApplication().launch() with continueAfterFailure = false. A slow simulator turns a would-be pass into a failure. This became reachable when fixtures started being regenerated on every run instead of downloaded as fixed artifacts. Exact per-bucket counts were safe against byte-identical fixtures; they are not against freshly generated ones. Assert what the sample sources actually determine — total, skipped, mixed, the bucket-sum invariant, and a floor on deliberate failures — and stop asserting the pass/fail split, which measures the simulator rather than this project. A lost test still breaks the "all == 13" assertion, dropped failures still break "failed >= 5", and miscounting still breaks the sum. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tylervick
force-pushed
the
fix-flaky-status-count
branch
from
August 7, 2026 20:24
9d9f916 to
895611c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
Codecovrun onmainis failing.Teston the same commit passes.Both run
./prepareTestResults.sh && swift test. Same code, same commit, different outcome — so this is nondeterminism, not a code defect.What happens
Passedwent 7→6 andFailedwent 5→6; the total stayed 13. One test moved buckets. The fixture-generation log showsFirstSuite.testOneas both passed and failed:Its body is
XCTAssert(true)plus an attachment — it cannot fail on its own assertion. The failure comes fromsetUp, which callsXCUIApplication().launch()withcontinueAfterFailure = false. Under CI load the simulator is sometimes slow to launch the app, and the test fails before its body runs.Why it started now
This is a consequence of #379. Fixtures used to be fixed artifacts downloaded from a private bucket — byte-identical every run, so exact per-bucket assertions were safe. They are now regenerated on every run, which is a better trade overall (they can't go stale, and fork PRs can run them), but it means UI-test flakiness feeds straight into assertions that were written for stable input.
The fix
Assert only what the sample sources actually determine:
all == 13skipped == 1SampleAppUnitTests.testSkippedis an unconditionalXCTSkipIfmixed == 0TestResultsexcludesRetryTestspassed + failed == all - skippedfailed >= 5The exact pass/fail split is dropped, because it measures whether the simulator launched an app in time — not anything this project does.
This still catches real regressions. Losing a test breaks
all == 13. Silently dropping failures breaksfailed >= 5. Miscounting breaks the sum invariant. Under the flaked run (passed=6, failed=6) the new assertions correctly pass.testMixedStatusis left alone: it readsRetryResults, andRetryTestshas nosetUpand never launches the app, so it isn't exposed to this.Follow-up
The underlying flakiness is worth its own issue — the sample UI tests launch the app purely so the run produces screen recordings (the only
.mp4fixture coverage), and none of them actually interact with the UI. There may be a way to keep the recordings without the launch being a failure point.🤖 Generated with Claude Code