toolchain-drift/test fixture verify gate accepts stub bundles — assert non-empty test data · Issue #454 · XCTestHTMLReport/XCTestHTMLReport · GitHub
Skip to content

toolchain-drift/test fixture verify gate accepts stub bundles — assert non-empty test data #454

Closed

Description

@tylervick

Found during the PR #453 review. A CI runner's simulator vanished mid-prepareTestResults.sh (Unable to find a device matching the provided destination specifier), the script's || true swallowed the failure, and the resulting stub .xcresult (zero testNodes; neither backend can render it) passed the fixture verify step, which checks only Info.plist presence. The test job then failed downstream with a confusing error instead of failing fast at generation.

Fix: harden the verify step (shared by test.yml and toolchain-drift.yml via the fixture cache path) to assert each bundle actually contains test data — e.g. xcrun xcresulttool get test-results summary reporting a non-zero totalTestCount, per bundle. That converts a runner-infrastructure flake into an immediate, attributable failure and keeps a stub from ever being SAVED to the fixture cache (the near-miss scenario: a stub cached under a valid key would poison every subsequent hit until the key rotates).

Two smaller notes from the same review, fine to fold in here or fix opportunistically: scope the 'no other fixture test produces Expected Failure' doc comment in the sample app to TestResults.xcresult (RetryTests.testInUnknownState already records one in RetryResults), and consider pinning renderer scale=1 for the programmatic PNG attachment so it is truly 8×8.

Refs #453 (review), #412/#436 (cache), #392/#442 (drift workflow).

Activity

  1. added this to the 4.0 milestone on Aug 13, 2026
  2. added 2 commits that reference this issue on Aug 14, 2026
    dfff3d5
    b038c76
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions