Migrate off xcresulttool --legacy before Apple removes it

This is the existential risk for this project.

XCResultKit drives xcrun xcresulttool get --format json --legacy. Apple has deprecated the legacy commands and will eventually remove them; when that happens, xchtmlreport stops working entirely.

We are two hops from a change we do not control: upstream davidahouse/XCResultKit last shipped v1.2.2 in Feb 2025, and its own tracking issue (davidahouse/XCResultKit#58, "Stop using --legacy") is open and unaddressed.

Task: decide the strategy and then execute it:

contribute the migration upstream to XCResultKit (best for the ecosystem, slowest, depends on upstream responsiveness), or
vendor/fork XCResultKit into this repo (full control, ongoing maintenance cost), or
read the new xcresulttool JSON format directly and drop the dependency

Whichever path, the fixture suite added in 3.0 is what makes the migration verifiable — the new format can be diffed against the old one on the same bundles.

Add Swift Testing fixture and assertions (#417)

Research findings, recorded here so they outlive the branch

A design spec and implementation plan were produced on tylervick/legacy-migration-391 (PR #426). The plan is not being merged — it is execution scaffolding that would go stale. The findings below are not, because they were measured rather than reasoned, against Xcode 26.2 (17C52), xcresulttool 24514, schema 0.1.0, legacy format 3.56.

Recommended strategy: read the new format directly, drop XCResultKit

Behind a reader abstraction that keeps the legacy path alive until Apple removes it. Not yet ratified — recorded as the researched recommendation.

Both alternatives were rejected for concrete reasons:

Contribute upstream — XCResultKit's public API is the legacy object graph (ActionsInvocationRecord, ActionTestSummaryGroup are one-to-one with legacy JSON types). Migrating it means proposing a breaking API redesign to a maintainer who has not responded to the tracking issue in over a year, and blocking an existential risk on a third party.
Vendor/fork — solves "upstream is unresponsive" but not "the format is being removed"; the vendored code still shells out to --legacy. Inherits 3,549 lines across 78 files to use ~15 types, and the new-format reader still has to be written.

Since all three options require writing a new-format reader, the direct read is strictly less work for the same outcome.

The central finding: the new format is not a superset

Byte-identical output across backends is therefore not achievable. The bar has to be a declared, reviewed diff asserted in CI — not an empty one. Anyone who plans this work expecting parity will be wrong.

Tree shape changes

LEGACY  Target → "All tests" → "Target.xctest" → Target → tests
                → "SwiftTestingSuite" → tests        (sibling of "All tests")
MODERN  Target → Target → tests
                → SwiftTestingSuite → tests          (consistent siblings)

Synthesizing the legacy wrappers was rejected: "Selected tests" vs "All tests" depends on run filtering, which the new format does not expose, so one label would always be fabricated. Swift Testing display names also differ — legacy taggedMultiplication() vs modern Tagged multiplication check.

Attachments get materially better

xcresulttool export attachments --path B --output-path D exports every attachment in one call (0.05s on TestResults), writing <uuid>.<ext> plus a manifest.json. The join is activities[].attachments[].uuid ↔ basename of manifest[].attachments[].exportedFileName, verified concretely (4DB9AD3F-… ↔ 4DB9AD3F-….mp4).

This replaces the legacy one-subprocess-per-payload export — and with it the per-payload lock table in ResultFile that exists only because XCResultKit exports every caller of an id to one shared temp path.

Performance: no claim in either direction

Same-runner interleaved A/B, 5 reps each: legacy 27 subprocess spawns / 0.74s median; modern 20 spawns / 0.76s median. Equal within noise. Measured only to rule out the modern path being dramatically worse because activities is per-test-id. Do not cite as a speedup, and do not compare against numbers from another runner.

Full spec and task-by-task plan remain on tylervick/legacy-migration-391 / PR #426 if the work is picked up.

docs: retarget the migration at 4.0 and rebase it onto #430

docs: design spec for migrating off xcresulttool --legacy (#391)

docs: implementation plan for the xcresulttool legacy migration (#391)

docs: retarget the migration at 4.0 and rebase it onto #430

Settle the ParsedResult information model before the port (#391) (#435)

Cross-backend identifier normalizer and baseline render capture (#391) (

XCResultToolClient and new-format schema structs (#391) (#441)

ci: add scheduled toolchain drift detector (#392)

The port: ParsedResult model, LegacyResultReader, and the renderer mi…

ci: scheduled Xcode/Swift toolchain drift detector (#392) (#442)

ModernResultReader, ModernPayloadStore, and extension-based attachmen…

test: guard the legacy arm on toolchain capability

Backend selection and the --result-reader flag (#391, Task 11) (#448)

THE DIFFERENTIAL: legacy/modern parity held to the declared allow-lis…

docs: state schema identity, not value identity, in the README

`--json` emits ParsedResult behind a documented wire contract + 4.0 r…

Upload the rendered report as a PR artifact, per reader backend (#457)

Summary initialiser taking pre-parsed runs, for fixture-free rendering

Run the browser assertions in CI (#461)

Closing as complete. Everything in this issue's scope is merged and proven on main:

The port + both readers: The port: ParsedResult model, LegacyResultReader, and the renderer migration (#391, Tasks 3–5b) #443 (ParsedResult + LegacyResultReader + renderer migration, byte-identity gated), XCResultToolClient and new-format schema structs (#391) #441 (XCResultToolClient + schema, pinned to schema-version 0.1.0), ModernResultReader, ModernPayloadStore, and extension-based attachment typing (#391, Tasks 8–10) #447 (ModernResultReader + ModernPayloadStore + both-backend attachment typing).
Backend selection: Backend selection and the --result-reader flag (#391, Task 11) #448 — --result-reader legacy|modern|auto, explicit legacy never silently substituted, demotion scoped to capability detection only.
The proof: THE DIFFERENTIAL: legacy/modern parity held to the declared allow-list + forced-modern CI leg (#391, Tasks 12–13) #450 — the differential runs on every PR in two CI legs (auto + forced-modern), holding legacy and modern renders equal outside a four-entry declared allow-list with anti-rot enforcement in both directions; structural counts, status maps, and attachment bytes asserted exactly.
--json on our own schema: --json emits ParsedResult behind a documented wire contract + 4.0 release notes (#391, Tasks 14–15) #451 — versioned wire contract at docs/json-schema.md, exportRecursiveJson() gone.
Early warning: ci: scheduled Xcode/Swift toolchain drift detector (#392) #442/ci: fix toolchain-drift false alarm — create --output dir, report exit code #445 — the scheduled toolchain-drift workflow alarms (via auto-filed issue) when Apple removes the legacy commands.

Fixed along the way: #449 (screen-recording export race, root-caused to Xcode's shared display names; exports are now content-addressed <payloadId>.<ext>). Breaking changes are documented in docs/release-notes/4.0.0.md.

Deliberately not part of this issue (per the design spec's phase 6): removing XCResultKit and LegacyResultReader waits until Apple actually removes --legacy — the differential keeps running for as long as both backends exist. Tracked in the follow-up issue filed alongside this close.

Watch the next Xcode from a beta drift leg that can never redden the …

Record what actually costs time when generating a report (#491)

Stop paying for subprocesses the reader never needed (refs #237) (#495)

Legacy	New format	Verdict
activity `activityType` (5 constants driving CSS classes)	absent; only `isAssociatedWithFailure`	lost
activity `start` and `finish`	`startTime` only	lost — per-activity duration not derivable
attachment `name` (user-supplied, e.g. `"HTML"`)	`name` holds the legacy filename	lost — survives only in a sibling child-activity title
attachment `uniformTypeIdentifier`	absent	derivable from `exportedFileName` extension
`ActionTestFailureSummary.fileName`/`.lineNumber`/`.issueType`	one string: `"RetryTests.swift:31: XCTAssertTrue failed"`	lost as structure, text preserved
log section `emittedOutput`	structured `messages`	lost
`exportRecursiveJson()` (drives `--json`)	none	no replacement
repetitions	first-class `nodeType: "Repetition"`	improved

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Migrate off xcresulttool --legacy before Apple removes it #391

Research findings, recorded here so they outlive the branch

Recommended strategy: read the new format directly, drop XCResultKit

The central finding: the new format is not a superset

Tree shape changes

Attachments get materially better

Performance: no claim in either direction

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

Migrate off xcresulttool --legacy before Apple removes it #391

Description

Activity

tylervick commented on Aug 11, 2026

Research findings, recorded here so they outlive the branch

Recommended strategy: read the new format directly, drop XCResultKit

The central finding: the new format is not a superset

Tree shape changes

Attachments get materially better

Performance: no claim in either direction

tylervick commented on Aug 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions