Repository navigation
fix(vitest): report browser retries and failure artifacts - #1604
formaceft-93 wants to merge 2 commits into
Conversation
6e283d1 to
b1317c6
Compare
b1317c6 to
ba55a35
Compare
todti
left a comment
There was a problem hiding this comment.
Thanks for the fix, the approach (snapshotting each attempt in onTestFailed and marking Retry as an excluded parameter so the attempts share a historyId) makes sense and the browser tests exercise the real Playwright flow.
There is one blocking problem (task.artifacts is dereferenced without a guard although the peer range reaches back to vitest 1.3) and a few behaviours that should be fixed or explicitly justified: the Retry parameter now also appears for node tests, and the trace lookup relies on vitest internals. Details are in the inline comments.
|
|
||
| task.meta.allureRuntimeMessages = []; | ||
| // Vitest keeps browser artifacts on the task across retries, so clear them after snapshotting the attempt. | ||
| task.artifacts.length = 0; |
There was a problem hiding this comment.
Blocking: task.artifacts.length = 0 is not guarded, while the line above (task.artifacts ? [...task.artifacts] : []) is. task.artifacts only exists in vitest 4, but the package declares "vitest": ">=1.3.0" as a peer dependency. On older versions this hook throws a TypeError on exactly the failing tests in browser mode. Please guard it (e.g. if (task.artifacts) { task.artifacts.length = 0; }) or narrow the peer range.
| const errors = task.result?.errors?.slice(attemptErrorOffset); | ||
|
|
||
| task.meta.allureFailedAttempts ??= []; | ||
| task.meta.allureFailedAttempts.push({ |
There was a problem hiding this comment.
Two remarks about what is stored in task.meta:
allureFailedAttemptskeeps a copy of the whole result, runtime messages and artifacts (includingbody: Uint8Arrayof screenshots).task.metais synchronized from the browser to node on task updates, so the heavy bodies get duplicated and sent again. Storing paths and light metadata would be enough.allureAttemptDurationis only set inonTestFinished, which runs afteronTestFailed, so the?? Date.now() - attemptStartTimefallback above is what is actually used for failed attempts, while passed attempts use theonTestFinishedvalue. Please add a short comment about this, otherwise the two durations look inconsistent by accident.
| } | ||
| } | ||
|
|
||
| if ((taskResult?.retryCount ?? 0) > 0) { |
There was a problem hiding this comment.
writeTaskAttempt is shared with node-mode tests, so every node test with retryCount > 0 now gets a Retry parameter too. That is a behaviour change for non-browser reporting that is not mentioned in the PR description and not covered by a test. Either limit it to browser tests or state it explicitly and add a node-mode test (there is a vitestTestEnvironments helper for running a test in both environments).
| const retryCount = result.retryCount ?? 0; | ||
| const traceDirectory = join(dirname(testFilePath), "__traces__", basename(testFilePath)); | ||
| // Vitest browser trace names are deterministic but internal; prefer its own full test name formatting. | ||
| const traceFileNames = [ |
There was a problem hiding this comment.
The trace lookup depends on vitest internals: the hardcoded __traces__ directory next to the test file and the reverse-engineered file name (project-testname-repeat-retry). The comment above already says the format is internal. If the user configures a custom traces directory, or the naming changes in a minor vitest release, the trace will silently stop being attached.
Please prefer taking the path from vitest itself (the task artifacts / browser config) and fall back to this guess only if needed. Also, why are two name variants needed (getTestName(task, "-") and getTestMetadata(task).name)? A comment on when each one applies would help.
| var allureTestPlan: TestPlanV1 | undefined; | ||
| } | ||
|
|
||
| type AllureVitestAttempt = { |
There was a problem hiding this comment.
AllureVitestAttempt is declared here (with unknown fields) and again in reporter.ts, which is why allureFailedAttempts as AllureVitestAttempt[] cast is needed. Consider declaring the type once and importing it in both places.
| it("should report browser retries as separate test results", async () => { | ||
| const { tests, attachments } = await runVitestInlineTest({ | ||
| "vitest.config.ts": ({ allureResultsPath }) => ` | ||
| ${createVitestBrowserConfig(allureResultsPath) |
There was a problem hiding this comment.
The config is patched with String.replace on the text returned by createVitestBrowserConfig. If the template changes, the replacement silently becomes a no-op and the test fails for a confusing reason. It would be more robust to extend createVitestBrowserConfig with options (retry, screenshotFailures, trace).
Also missing from the tests:
- steps/labels/attachments from a failed attempt must not leak into the following passed attempt;
- a passed attempt after a failed one must have no trace/screenshot;
Retryparameter behaviour in node mode (see the comment onreporter.ts).
ba55a35 to
1f06313
Compare
todti
left a comment
There was a problem hiding this comment.
A few inline notes on retry handling and artifact snapshotting.
|
|
||
| const attempts: AllureVitestAttempt[] = [ | ||
| ...(allureFailedAttempts as AllureVitestAttempt[]), | ||
| ...(task.result?.state === "fail" && allureFailedAttempts.length ? [] : [{}]), |
There was a problem hiding this comment.
When the final state is fail and allureFailedAttempts is non-empty, the final attempt is skipped on the assumption that onTestFailed already snapshotted it. If a failure appears after onTestFailed ran (for example an error thrown in an afterEach hook), that last result is never written. Could we always write the final attempt and only dedupe when the last snapshot is known to be the same attempt?
There was a problem hiding this comment.
I changed the lifecycle so earlier attempts are finalized at the next beforeEach, after their callbacks and teardown have completed, and the reporter always writes the authoritative final task result. This removes the blanket “failed snapshots exist, so skip the final result” condition without duplicating the final attempt. I added browser regressions for a late aroundEach teardown failure and for both final test.fails outcomes after retries.
| ...task.result, | ||
| startTime: attemptStartTime, | ||
| // onTestFinished runs after onTestFailed, so failed attempts normally use this fallback. | ||
| duration: task.meta.allureAttemptDuration ?? Date.now() - attemptStartTime, |
There was a problem hiding this comment.
As the comment says, onTestFinished runs after onTestFailed, so for failed attempts the duration is measured at the moment of failure and excludes afterEach hooks. For the final attempt the reporter relies on allureAttemptDuration being synced to the node side, and otherwise falls back to task.result.duration, which may span all retries. Worth double-checking that fallback can't produce inflated durations.
There was a problem hiding this comment.
I checked the runner ordering: afterEach → onTestFinished → onTestFailed in Vitest 2.1.9, 3.2.4 and 4.1.10. The comment in my code had this backwards; I corrected it. I also removed the cumulative task.result.duration fallback for browser attempts: they use their own timing metadata. The browser regression includes a delayed afterEach and verifies that its time is included in both attempt durations. Runner source.
| }); | ||
|
|
||
| onTestFailed(async ({ task }) => { | ||
| const errors = task.result?.errors?.slice(attemptErrorOffset); |
There was a problem hiding this comment.
attemptErrorOffset assumes task.result.errors accumulates across retries. If Vitest resets the array on retry, the offset from the previous attempt can be larger than the new array, so slice returns [] and the failed attempt loses its error (and therefore message/status). Is the accumulation behaviour guaranteed across supported Vitest versions?
There was a problem hiding this comment.
I checked the runner implementations in 1.3.0, 2.1.9, 3.2.4 and 4.1.10: normal retries append to result.errors rather than resetting it, so I kept the offset for that path. There is a separate replacement at finalization for test.fails; I now preserve that final error array directly and added browser regressions for expected failure and unexpected pass after retries. This avoids slicing away the “Expect test to fail” error.
| artifacts: toLightweightArtifacts(artifacts), | ||
| }); | ||
|
|
||
| task.meta.allureRuntimeMessages = []; |
There was a problem hiding this comment.
Runtime messages are snapshotted and reset here, but messages emitted afterwards by afterEach hooks of the failed attempt will end up on the next attempt (or the final result) instead of the one they belong to.
There was a problem hiding this comment.
Regular afterEach hooks actually finish before onTestFinished and onTestFailed, but later callbacks/teardown were a real gap. I moved finalization of earlier attempts to the next beforeEach and only reset the runtime-message buffer at that boundary. The final attempt is read from the completed task. The new browser regression checks that late teardown labels, steps and attachment messages stay on their own attempt.
| task.meta.allureRuntimeMessages = []; | ||
| if (artifacts) { | ||
| // Vitest keeps browser artifacts on the task across retries, so clear them after snapshotting the attempt. | ||
| artifacts.length = 0; |
There was a problem hiding this comment.
artifacts.length = 0 mutates Vitest's own task.artifacts, which other reporters (html/json/UI) also read, so they will lose failure screenshots/traces of earlier attempts when the Allure reporter is enabled. Could we track an index/offset of already-consumed artifacts instead of clearing the shared array?
There was a problem hiding this comment.
I removed the mutation and now track artifactStart/artifactEnd offsets instead. This also removes the need to copy artifact bodies into task.meta. I added a regression that checks the browser can still see the previous screenshot on retry and a second reporter can read it at run end. One detail from checking the transport: node reporters already receive artifacts through a separate RPC, so I could not reproduce their loss on the old code; the browser-side array mutation was reproducibly observable.
| } | ||
|
|
||
| private writeTaskArtifacts(testUuid: string, taskResult: TaskResultLike | undefined, artifacts: VitestArtifact[]) { | ||
| if (taskResult?.state !== "fail") { |
There was a problem hiding this comment.
Screenshots (and the trace below) are attached only when the attempt state is fail. Artifacts produced by tests that end up passing (e.g. toMatchScreenshot diffs, or trace: "on") are never attached. Is that intended?
There was a problem hiding this comment.
That restriction was not intended. I removed the final-status guards from artifact and trace attachment. Available screenshot-comparison artifacts are now attached for passed tests too, and trace: "on" attaches a separate trace to both the failed attempt and the passed retry. Added browser regressions for both cases, including a custom tracesDir, and verified that the failure screenshot does not leak into the passed retry.
| const retryCount = result.retryCount ?? 0; | ||
| const traceDirectory = join(dirname(testFilePath), "__traces__", basename(testFilePath)); | ||
| // Fallback for Vitest versions/providers that write traces but don't expose them as annotation artifacts. | ||
| const traceFileNames = [ |
There was a problem hiding this comment.
traceFileNames is a one-element array iterated with a loop. This can be a single join(...) + existsSync check.
There was a problem hiding this comment.
I removed this fallback altogether rather than just simplifying its loop. Trace paths now come from the completed task’s annotations, where Vitest publishes them after all retries. The repeat/retry suffix in the annotation message is used only to select the matching attempt; the attachment path itself comes from Vitest. The custom-directory regression covers this path.
| import { fileURLToPath } from "node:url"; | ||
|
|
||
| import { Stage, Status } from "allure-js-commons"; | ||
| import { getTestName } from "@vitest/runner/utils"; |
There was a problem hiding this comment.
@vitest/runner/utils is a new direct import. Is @vitest/runner declared in this package's dependencies/peerDependencies? Otherwise it may fail to resolve under strict installs (pnpm / Yarn PnP) and break the reporter for every Vitest run, not only browser ones.
There was a problem hiding this comment.
@vitest/runner is already declared in both peerDependencies (>=1.3.0) and devDependencies on this branch. I also removed the new @vitest/runner/utils runtime import: it is no longer needed now that trace attachment paths come from Vitest’s annotations instead of being constructed locally.
todti
left a comment
There was a problem hiding this comment.
Thanks for the updates, the earlier points look addressed. One remaining note on trace matching.
There was a problem hiding this comment.
When there are several attempts, the trace is chosen by message.endsWith("-<repeat>-<retry>.trace.zip"), which depends on the exact format of Vitest's trace annotation message. If that format changes (or a custom naming scheme is used), nothing matches for retried tests and traces are silently dropped for every attempt. Could we fall back to the annotation attachment.path (for example by order, or by matching the suffix on the path as well as the message), or at least cover the format with a test so a Vitest upgrade fails loudly?

Summary
Fixes Vitest browser reporting so retry attempts are exposed as separate Allure results instead of only showing the last attempt.
fixes #1451
Checklist