fix(utils): fail tests for severe test hook errors by carrerasdarren-cell · Pull Request #16110 · webdriverio/webdriverio · GitHub
Skip to content

fix(utils): fail tests for severe test hook errors - #16110

Open
carrerasdarren-cell wants to merge 2 commits into
webdriverio:mainfrom
carrerasdarren-cell:codex/severe-test-hooks-14314-20261017
Open

carrerasdarren-cell wants to merge 2 commits into
webdriverio:mainfrom
carrerasdarren-cell:codex/severe-test-hooks-14314-20261017

Conversation

@carrerasdarren-cell

@carrerasdarren-cell carrerasdarren-cell commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Proposed changes

Fixes #14314.

A SevereServiceError from Mocha or Jasmine beforeTest/afterTest currently logs an error while the affected test passes. Fail that test through the existing shared test wrapper. A severe beforeTest error prevents its body from running and still reaches afterTest with the failure; a severe afterTest error fails an otherwise passing test. Ordinary hook errors, existing body failures, skips, later tests, and Mocha native retries retain their behavior. The check scans all hook results, so an earlier ordinary error cannot hide a later severe error.

This is limited to Test wrappers; Cucumber's separate scenario/step hooks and the global hook executor are unchanged. Document the test-level behavior and add a dedicated named mock-driver smoke suite.

How you tested

  • Compiled clean main 795fa22a0ade22cdb84dfbece5c57926fd981203 before the baseline. Actual Mocha/Jasmine Launcher runs and JSON reports reproduced four false-green severe cases among ten baseline cases; all ten meet the intended behavior after the change. Six ordinary/no-error controls retain their results.

  • Added durable regressions before editing product code: two fail on the original behavior; the focused file then passes all 21 tests.

  • pnpm run test:package wdio-utils, wdio-mocha-framework, and wdio-jasmine-framework: 725 tests pass across 33 files, with no type errors.

  • pnpm run test:smoke severeTestHooks: 18 case assertions pass with actual Launcher/reporter events, covering ordinary and severe errors, mixed hook arrays, subsequent tests, original body/expectation failures, and successful one-shot Mocha native retries. These use the repository's mock driver, not a real browser.

  • Review follow-up: two additional regressions fail on the initially submitted revision and the focused file now passes 25 tests. A newly severe afterTest failure pauses the debug agent once after that hook; local broker/CLI checks cover resume and close for both frameworks and preserve the existing body-failure pause. All six runtime cases pass with the mock driver. Jasmine retains its existing adapter-provided bundled spec identity; no identity metadata repair is claimed. The smoke suite also removes only its own temporary directory, verified on success and an intentional failure path while preserving prior artifacts.

  • Existing named smokes pass: mochaHooksTestrunner, jasmineHooksTestrunner, jasmineAfterHookArgsValidation, severeErrorTest, and cucumberTestrunner.

  • pnpm run test:typings:mocha, pnpm run test:typings:jasmine, and pnpm run test:oxlint pass.

  • pnpm run docs:generate:en and pnpm --filter=@wdio/docs run check --skip-build-checks pass. The full docs:check requires an absent website/build; no production site build was run.

  • Unit tests for the touched package(s)

  • Type definition tests

  • Smoke or e2e

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • Documentation update

Checklist

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective
  • I have added the necessary documentation
  • I have added proper type definitions for new commands (no new commands or public types)

Backport Request

This change targets current main/v10. No v9 backport is included.

Further comments

Codex assisted with implementation, review, and the local validation listed above. Mocha native framework retries are tested; this does not claim that WDIO's body-only repeatTest retries rerun test hooks.

@carrerasdarren-cell
carrerasdarren-cell requested a review from a team as a code owner October 17, 2026 03:01
@wdio-bot wdio-bot added the PR: Bug Fix 🐛 PRs that contain bug fixes label Oct 17, 2026
@greptile-apps

greptile-apps Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Comment on lines +220 to +221
if (type === 'Test' && !error && !skip) {
error = getSevereError(afterResults)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Failed test does not pause

A severe afterTest error now fails the test, but it is assigned after the only call to getDebugAgentPause. With --debug=agent, an otherwise passing test that fails here moves on without pausing for inspection. Pause on this new failure path too, and add a regression test.

Comment thread tests/smoke.runner.js
Comment on lines +945 to +955

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Smoke runs leave files behind

severeTestHooks creates a fresh temporary directory on every run but never removes it. Each run leaves its reports and worker logs behind. Remove directory in finally after restoring the environment variables.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR: Bug Fix 🐛 PRs that contain bug fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[💡 Feature]: Ability to set the test status from the afterTest hook

2 participants