fix(vitest): report browser retries and failure artifacts by formaceft-93 · Pull Request #1604 · allure-framework/allure-js · GitHub
Skip to content

fix(vitest): report browser retries and failure artifacts - #1604

Open
formaceft-93 wants to merge 2 commits into
mainfrom
fix-vitest-browser-retries-artifacts
Open

formaceft-93 wants to merge 2 commits into
mainfrom
fix-vitest-browser-retries-artifacts

Conversation

@formaceft-93

@formaceft-93 formaceft-93 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes Vitest browser reporting so retry attempts are exposed as separate Allure results instead of only showing the last attempt.

fixes #1451

Checklist

@formaceft-93
formaceft-93 requested a review from epszaw October 2, 2026 09:31
@formaceft-93 formaceft-93 added the pr:bug Pull requests that fix broken or incorrect behavior. label Oct 2, 2026
@formaceft-93
formaceft-93 force-pushed the fix-vitest-browser-retries-artifacts branch from 6e283d1 to b1317c6 Compare October 2, 2026 10:28
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

@formaceft-93
formaceft-93 force-pushed the fix-vitest-browser-retries-artifacts branch from b1317c6 to ba55a35 Compare October 12, 2026 07:16
@formaceft-93
formaceft-93 requested a review from todti October 13, 2026 10:41

@todti todti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix, the approach (snapshotting each attempt in onTestFailed and marking Retry as an excluded parameter so the attempts share a historyId) makes sense and the browser tests exercise the real Playwright flow.

There is one blocking problem (task.artifacts is dereferenced without a guard although the peer range reaches back to vitest 1.3) and a few behaviours that should be fixed or explicitly justified: the Retry parameter now also appears for node tests, and the trace lookup relies on vitest internals. Details are in the inline comments.


task.meta.allureRuntimeMessages = [];
// Vitest keeps browser artifacts on the task across retries, so clear them after snapshotting the attempt.
task.artifacts.length = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: task.artifacts.length = 0 is not guarded, while the line above (task.artifacts ? [...task.artifacts] : []) is. task.artifacts only exists in vitest 4, but the package declares "vitest": ">=1.3.0" as a peer dependency. On older versions this hook throws a TypeError on exactly the failing tests in browser mode. Please guard it (e.g. if (task.artifacts) { task.artifacts.length = 0; }) or narrow the peer range.

const errors = task.result?.errors?.slice(attemptErrorOffset);

task.meta.allureFailedAttempts ??= [];
task.meta.allureFailedAttempts.push({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two remarks about what is stored in task.meta:

  1. allureFailedAttempts keeps a copy of the whole result, runtime messages and artifacts (including body: Uint8Array of screenshots). task.meta is synchronized from the browser to node on task updates, so the heavy bodies get duplicated and sent again. Storing paths and light metadata would be enough.
  2. allureAttemptDuration is only set in onTestFinished, which runs after onTestFailed, so the ?? Date.now() - attemptStartTime fallback above is what is actually used for failed attempts, while passed attempts use the onTestFinished value. Please add a short comment about this, otherwise the two durations look inconsistent by accident.

Comment thread packages/allure-vitest/src/reporter.ts Outdated
}
}

if ((taskResult?.retryCount ?? 0) > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

writeTaskAttempt is shared with node-mode tests, so every node test with retryCount > 0 now gets a Retry parameter too. That is a behaviour change for non-browser reporting that is not mentioned in the PR description and not covered by a test. Either limit it to browser tests or state it explicitly and add a node-mode test (there is a vitestTestEnvironments helper for running a test in both environments).

Comment thread packages/allure-vitest/src/reporter.ts Outdated
const retryCount = result.retryCount ?? 0;
const traceDirectory = join(dirname(testFilePath), "__traces__", basename(testFilePath));
// Vitest browser trace names are deterministic but internal; prefer its own full test name formatting.
const traceFileNames = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trace lookup depends on vitest internals: the hardcoded __traces__ directory next to the test file and the reverse-engineered file name (project-testname-repeat-retry). The comment above already says the format is internal. If the user configures a custom traces directory, or the naming changes in a minor vitest release, the trace will silently stop being attached.

Please prefer taking the path from vitest itself (the task artifacts / browser config) and fall back to this guess only if needed. Also, why are two name variants needed (getTestName(task, "-") and getTestMetadata(task).name)? A comment on when each one applies would help.

var allureTestPlan: TestPlanV1 | undefined;
}

type AllureVitestAttempt = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AllureVitestAttempt is declared here (with unknown fields) and again in reporter.ts, which is why allureFailedAttempts as AllureVitestAttempt[] cast is needed. Consider declaring the type once and importing it in both places.

it("should report browser retries as separate test results", async () => {
const { tests, attachments } = await runVitestInlineTest({
"vitest.config.ts": ({ allureResultsPath }) => `
${createVitestBrowserConfig(allureResultsPath)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config is patched with String.replace on the text returned by createVitestBrowserConfig. If the template changes, the replacement silently becomes a no-op and the test fails for a confusing reason. It would be more robust to extend createVitestBrowserConfig with options (retry, screenshotFailures, trace).

Also missing from the tests:

  • steps/labels/attachments from a failed attempt must not leak into the following passed attempt;
  • a passed attempt after a failed one must have no trace/screenshot;
  • Retry parameter behaviour in node mode (see the comment on reporter.ts).

@formaceft-93
formaceft-93 force-pushed the fix-vitest-browser-retries-artifacts branch from ba55a35 to 1f06313 Compare October 13, 2026 12:37
@formaceft-93
formaceft-93 requested a review from todti October 13, 2026 16:38

@todti todti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few inline notes on retry handling and artifact snapshotting.

Comment thread packages/allure-vitest/src/reporter.ts Outdated

const attempts: AllureVitestAttempt[] = [
...(allureFailedAttempts as AllureVitestAttempt[]),
...(task.result?.state === "fail" && allureFailedAttempts.length ? [] : [{}]),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the final state is fail and allureFailedAttempts is non-empty, the final attempt is skipped on the assumption that onTestFailed already snapshotted it. If a failure appears after onTestFailed ran (for example an error thrown in an afterEach hook), that last result is never written. Could we always write the final attempt and only dedupe when the last snapshot is known to be the same attempt?

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the lifecycle so earlier attempts are finalized at the next beforeEach, after their callbacks and teardown have completed, and the reporter always writes the authoritative final task result. This removes the blanket “failed snapshots exist, so skip the final result” condition without duplicating the final attempt. I added browser regressions for a late aroundEach teardown failure and for both final test.fails outcomes after retries.

...task.result,
startTime: attemptStartTime,
// onTestFinished runs after onTestFailed, so failed attempts normally use this fallback.
duration: task.meta.allureAttemptDuration ?? Date.now() - attemptStartTime,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the comment says, onTestFinished runs after onTestFailed, so for failed attempts the duration is measured at the moment of failure and excludes afterEach hooks. For the final attempt the reporter relies on allureAttemptDuration being synced to the node side, and otherwise falls back to task.result.duration, which may span all retries. Worth double-checking that fallback can't produce inflated durations.

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the runner ordering: afterEach → onTestFinished → onTestFailed in Vitest 2.1.9, 3.2.4 and 4.1.10. The comment in my code had this backwards; I corrected it. I also removed the cumulative task.result.duration fallback for browser attempts: they use their own timing metadata. The browser regression includes a delayed afterEach and verifies that its time is included in both attempt durations. Runner source.

});

onTestFailed(async ({ task }) => {
const errors = task.result?.errors?.slice(attemptErrorOffset);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attemptErrorOffset assumes task.result.errors accumulates across retries. If Vitest resets the array on retry, the offset from the previous attempt can be larger than the new array, so slice returns [] and the failed attempt loses its error (and therefore message/status). Is the accumulation behaviour guaranteed across supported Vitest versions?

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the runner implementations in 1.3.0, 2.1.9, 3.2.4 and 4.1.10: normal retries append to result.errors rather than resetting it, so I kept the offset for that path. There is a separate replacement at finalization for test.fails; I now preserve that final error array directly and added browser regressions for expected failure and unexpected pass after retries. This avoids slicing away the “Expect test to fail” error.

artifacts: toLightweightArtifacts(artifacts),
});

task.meta.allureRuntimeMessages = [];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runtime messages are snapshotted and reset here, but messages emitted afterwards by afterEach hooks of the failed attempt will end up on the next attempt (or the final result) instead of the one they belong to.

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regular afterEach hooks actually finish before onTestFinished and onTestFailed, but later callbacks/teardown were a real gap. I moved finalization of earlier attempts to the next beforeEach and only reset the runtime-message buffer at that boundary. The final attempt is read from the completed task. The new browser regression checks that late teardown labels, steps and attachment messages stay on their own attempt.

task.meta.allureRuntimeMessages = [];
if (artifacts) {
// Vitest keeps browser artifacts on the task across retries, so clear them after snapshotting the attempt.
artifacts.length = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

artifacts.length = 0 mutates Vitest's own task.artifacts, which other reporters (html/json/UI) also read, so they will lose failure screenshots/traces of earlier attempts when the Allure reporter is enabled. Could we track an index/offset of already-consumed artifacts instead of clearing the shared array?

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the mutation and now track artifactStart/artifactEnd offsets instead. This also removes the need to copy artifact bodies into task.meta. I added a regression that checks the browser can still see the previous screenshot on retry and a second reporter can read it at run end. One detail from checking the transport: node reporters already receive artifacts through a separate RPC, so I could not reproduce their loss on the old code; the browser-side array mutation was reproducibly observable.

Comment thread packages/allure-vitest/src/reporter.ts Outdated
}

private writeTaskArtifacts(testUuid: string, taskResult: TaskResultLike | undefined, artifacts: VitestArtifact[]) {
if (taskResult?.state !== "fail") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshots (and the trace below) are attached only when the attempt state is fail. Artifacts produced by tests that end up passing (e.g. toMatchScreenshot diffs, or trace: "on") are never attached. Is that intended?

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That restriction was not intended. I removed the final-status guards from artifact and trace attachment. Available screenshot-comparison artifacts are now attached for passed tests too, and trace: "on" attaches a separate trace to both the failed attempt and the passed retry. Added browser regressions for both cases, including a custom tracesDir, and verified that the failure screenshot does not leak into the passed retry.

Comment thread packages/allure-vitest/src/reporter.ts Outdated
const retryCount = result.retryCount ?? 0;
const traceDirectory = join(dirname(testFilePath), "__traces__", basename(testFilePath));
// Fallback for Vitest versions/providers that write traces but don't expose them as annotation artifacts.
const traceFileNames = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

traceFileNames is a one-element array iterated with a loop. This can be a single join(...) + existsSync check.

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this fallback altogether rather than just simplifying its loop. Trace paths now come from the completed task’s annotations, where Vitest publishes them after all retries. The repeat/retry suffix in the annotation message is used only to select the matching attempt; the attachment path itself comes from Vitest. The custom-directory regression covers this path.

Comment thread packages/allure-vitest/src/reporter.ts Outdated
import { fileURLToPath } from "node:url";

import { Stage, Status } from "allure-js-commons";
import { getTestName } from "@vitest/runner/utils";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vitest/runner/utils is a new direct import. Is @vitest/runner declared in this package's dependencies/peerDependencies? Otherwise it may fail to resolve under strict installs (pnpm / Yarn PnP) and break the reporter for every Vitest run, not only browser ones.

@formaceft-93 formaceft-93 Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vitest/runner is already declared in both peerDependencies (>=1.3.0) and devDependencies on this branch. I also removed the new @vitest/runner/utils runtime import: it is no longer needed now that trace attachment paths come from Vitest’s annotations instead of being constructed locally.

@formaceft-93
formaceft-93 requested a review from todti October 16, 2026 11:43

@todti todti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, the earlier points look addressed. One remaining note on trace matching.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When there are several attempts, the trace is chosen by message.endsWith("-<repeat>-<retry>.trace.zip"), which depends on the exact format of Vitest's trace annotation message. If that format changes (or a custom naming scheme is used), nothing matches for retried tests and traces are silently dropped for every attempt. Could we fall back to the annotation attachment.path (for example by order, or by matching the suffix on the path as well as the message), or at least cover the format with a test so a Vitest upgrade fails loudly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr:bug Pull requests that fix broken or incorrect behavior.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vitest browser retry support + traces and screenshot automatic attachment on fail

2 participants