Parameterized Swift Testing durations diverge across backends (legacy sums the argument executions, modern does not) · Issue #477 · XCTestHTMLReport/XCTestHTMLReport · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parameterized Swift Testing durations diverge across backends (legacy sums the argument executions, modern does not) #477
A parameterized Swift Testing test case reports a different duration on each backend, and the difference is currently hidden by the durations allow-list entry.
Measured
On CI's TestResults.xcresult (downloaded from the failing run of #476 and reproduced locally), exactly one leaf test disagrees:
SwiftTestingSuite/parameterizedAddition(value:) legacy 0.001268s modern 0.000423s
Legacy is ~3× modern, and the test has three argument sets. Every other leaf — XCTest cases, the non-parameterized Swift Testing cases, both other fixtures — is bit-identical.
The same measurement on a faster machine gives 0.0 on both backends, which is why it did not appear during #476's local development and only surfaced in CI.
Root cause
This is the arguments form of a problem the migration design already solved for repetitions.
8 | Test-case duration sums repetitions? | Yes — keep today's behaviour | The modern Test Case node reports its own duration, which is not the sum … Summing in the renderer makes both backends agree by construction.
TestCase.duration is iterations.reduce(0) { $0 + $1.duration }, so repetitions converge by construction. Answer 6 added ParsedTestCase.arguments at the same time and noted it was "unexercised by fixtures" — so nobody applied answer 8's treatment to it. It is exercised now: legacy merges a parameterized case's argument executions into iterations and therefore sums them; the modern reader reports the parameterized node's own durationInSeconds and does not sum its Arguments children.
Candidate fix
Have the modern reader sum the per-argument execution durations for a parameterized case, if xcresulttool's Arguments nodes expose them — the answer-8 move, converging the two backends by construction rather than by mask.
If the format does not expose per-argument durations, the alternative is the other direction the design prefers: hold legacy down to modern's capability, i.e. stop summing for the arguments case. Either way the divergence goes away instead of being masked.
Why it needs an issue rather than living under the mask
The spec is explicit that a mask is the worse of two outcomes:
Direction of the win: each answer either removes a field from the port or an entry from the differential allow-list. Deliberately holding the legacy backend down to the modern backend's capability makes the two agree, and an unmasked diff proves more than a masked one. The cost is one-way and honest … rather than a permanent asymmetry hidden behind a mask.
and differential-allowlist.json says an entry means "accepting a permanent difference between the backends — a design decision, not a way to quiet a failing test." No entry is being added here — this divergence already falls under the existing durations rule, and has since the parameterized fixture landed — but #476's summary header is the first place it aggregates into a single number, which is what made it visible. Filing it so the mask is a known debt with an owner rather than a rug.
When this is fixed
Two things come out, both named in the code:
the exclusion in DifferentialTests.testSummaryHeaderNumbersAgreeAcrossBackends, which today skips this one identifier from an otherwise exact per-leaf duration comparison;
the paragraph in Sources/XCTestHTMLReportCore/Classes/Models/RunSummary.swift explaining why the header's total is written parenthesised.
The header's duration can then be written in any shape, because it would no longer need the durations rule to cover it.
Found while implementing #439's summary header (#476).
Correction to this issue's body (found during PR #476's fix round): closing this issue does NOT free the summary header's duration from the masked '(N.NNs)' shape. A second, permanent property keeps the mask necessary: repetition durations are independently measured by the two formats and differ at µs scale per repetition (measured 0–376µs across three fixture generations; recorded in the migration spec's Verification section by #476). Fixing the parameterized summing here removes the structural 3× divergence only. The header duration therefore stays in the masked shape indefinitely.
Activity
Cover the header's duration with the known loss it actually inherits …
Stop asserting an exactness the formats never promised for repetition…
tylervick commentedon Aug 14, 2026
Correction to this issue's body (found during PR #476's fix round): closing this issue does NOT free the summary header's duration from the masked '(N.NNs)' shape. A second, permanent property keeps the mask necessary: repetition durations are independently measured by the two formats and differ at µs scale per repetition (measured 0–376µs across three fixture generations; recorded in the migration spec's Verification section by #476). Fixing the parameterized summing here removes the structural 3× divergence only. The header duration therefore stays in the masked shape indefinitely.
Give the report a summary header it can be triaged from (refs #439) (#…