fix(@wdio/cucumber-framework): report step results in parallel mode as in sequential mode by plum117 · Pull Request #16161 · webdriverio/webdriverio · GitHub
Skip to content

fix(@wdio/cucumber-framework): report step results in parallel mode as in sequential mode - #16161

Open
plum117 wants to merge 18 commits into
webdriverio:mainfrom
plum117:fix/cucumber-parallel-step-parity
Open

plum117 wants to merge 18 commits into
webdriverio:mainfrom
plum117:fix/cucumber-parallel-step-parity

Conversation

@plum117

@plum117 plum117 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With cucumberOpts.parallelMode, the parallel step path only converted the Cucumber status. The sequential formatter has more rules. All reporters got different step results in a parallel run:

# Parallel mode before Now (same as sequential)
G1 An undefined step is reported passed, with no error failed with Step "…" is not defined. …, or pending + (undefined step) with ignoreUndefinedDefinitions
G2 An ambiguous step is skipped with no reason; failAmbiguousDefinitions is ignored for steps skipped with the matching step definitions as pendingReason, or failed with failAmbiguousDefinitions
G3 A failed step's error message is the whole Cucumber message, so reporters show Error: Error: … and internal stack lines first line as message, Cucumber's message as stack
G4 cucumberOpts.order is ignored Cucumber's own orderPickles: random:<seed> gives the sequential order, an unknown order fails before any window opens
G5 A Before/After hook has an empty title hook name + tag expression, for example @beforeFails
G7 With tagsInTitle, a feature title has no tags; the feature start has no description and keyword @feat: Statuses in parallel, with description and keyword
G9 After a failFast stop, the scenarios that did not run are plain skipped steps, and are left out of the run result run in Cucumber's skip mode, with no window: hooks SKIPPED, an undefined step stays failed (and counts as a failure), an ambiguous step keeps its reason
G11 With order: 'random', the run uses another seed than the logged one (sequential), or logs 2 seeds (parallel) 1 seed, picked in init() and logged once; random:<logged seed> gives the same order again, in both modes
– A pending step had passed: true in its parallel payload (sequential: false) passed: false, as sequential
G10 --spec file.feature:12 fails with ENOENT (the file is read with the :12); with the file read, all its scenarios run only the scenarios at the line numbers run

G1 and G2 also apply with scenarioLevelReporter: an ambiguous scenario failed with no error or was skipped with no reason, and an ignored undefined scenario had no (undefined step) suffix. The scenario result now uses the same rule.

 cucumberFormatter.ts afterTest
-  status rules (undefined, failed, ambiguous, pendingReason, suffix)
+  getStepResult()            # utils.ts, shared
 parallel.ts onTestStepFinished
-  convertStatus(status), new Error(message)
+  getStepResult()            # same rules
+  ambiguous message from Cucumber's support code messages
 parallel.ts onTestStepStarted / onTestStepFinished (hooks)
-  text: ''
+  text: hook name + tag expression
 parallel.ts runParallelCucumber
+  orderPickles(allPickles, cucumberOpts.order)
 index.ts init
+  order 'random' → 'random:<seed>', logged once
   after a failFast stop
-    emitSkippedScenario(pickle)           # plain skipped steps
+    runScenario(pickle, no window)        # TestCaseRunner with skip: true

Docs: ParallelTests.md lists what parallel mode reports like a sequential run and the real differences; it and the spec reporter README now say that @wdio/spec-reporter does not support Cucumber parallel mode yet (as Allure and cucumberjs-json). The cucumberOpts docs (README, Frameworks.md, JSDoc) are corrected against the code: timeout/backtrace defaults, strict, retryTagFilter, ignoreUndefinedDefinitions, order, profiles, the tags session note, and the missing file, dryRun, worldParameters, scenarioLevelReporter, parallelMode, maxParallelContexts. Retry.md: Cucumber retry works without scenarioLevelReporter.

Evidence

Full matrix (headless Chrome, BiDi): 49 cases, each a sequential run and a parallel run of the same specs. Exit codes and the framework events of every scenario are the same, except the documented differences below. Cases: contexts/userContexts, 1/4/8 slots, fallback, ignoreUndefinedDefinitions × failAmbiguousDefinitions × strict (8), tagsInTitle, retry, retryTagFilter, order (random:7, random, rejected values), dryRun, failFast, tags, name, French, failed BeforeAll, line numbers, same file names, @skip(...), Rule/Background/Outline with tagged Examples/doc string/table/empty scenario/*/But, failing After, step timeout, BeforeStep/AfterStep, and the scenario-level variants.

Documented differences (ParallelTests.md): failFast lets the scenarios that already run finish; with ignoreUndefinedDefinitions: true, an ambiguous step (with failAmbiguousDefinitions: false) or a strict pending step fails the run (a sequential run then passes); Cucumber formatters write no file.

Real browser (headless Chrome, BiDi): the same 3 feature files run sequential, then with parallelMode: 'contexts' (4 slots). Each reporter's output is compared between the 2 runs.

Reporter Before After
spec S undefined ✓ passed; 2 failing instead of 3; Error: Error: fails on purpose + onTestStepFinished frames; failing hook shown as ✖ same as sequential
dot 2 F instead of 3 3 F
concise undefined step failure missing same as sequential
junit undefined step ✅ with no <failure>; failure message with the stack no diff
json undefined step passed; hook titles '' no diff
all (with failFast, 2 slots) spec 1 failing / 17 skipped (sequential 2 / 16); 6 of 8 scenarios with other events same as sequential

Also the same as sequential with ignoreUndefinedDefinitions: true, with failAmbiguousDefinitions: true, with scenarioLevelReporter: true (alone and with both options), with tagsInTitle (step and scenario level), with retry, dryRun, tags, name, userContexts, 1 and 8 slots, and with order: 'random:7' (scenario start order D tables, C, X, Y, D dropdown, A, B in both runs). With scenarioLevelReporter, JUnit puts the session-end DELETE /session command log under the last finished scenario, which is a different scenario in a parallel run; that is the reporter's rule for commands with no browsing context, not a step result.

  • Unit: 212 tests in @wdio/cucumber-framework (new: getStepResult, getFeatureStack, parallel step and scenario results, hook titles, order, the feature stack only for errors, feature titles, failFast skip mode, 1 random seed); the 2 older failFast tests now use a defined failing step, because skip mode keeps an undefined step failed; each new test failed before its fix.
  • Smoke: the 17 Cucumber suites pass (the formatter snapshots pass without edits).
  • Random seed (G11), headless Chrome, 7 scenarios: each run logs 1 seed; a run with random:<logged seed> gives the same scenario order (sequential random:64756, parallel random:550536), and 1 seed gives the same order in both modes.

Merge Danger

Door: two-way

Blast Radius: Cucumber parallel mode only (with failFast, the run result does not change: the run already failed). Sequential runs use the same rules as before, moved into getStepResult(); with order: 'random', a sequential run now logs the seed that it uses.

A parallel run with an undefined step now reports that step as failed. The run already failed before (the run result counted it), so no passing run starts to fail.

Out of scope: @wdio/allure-reporter, @wdio/spec-reporter and @wdio/cucumberjs-json-reporter still mix overlapping Cucumber scenarios in parallel mode (their own state, separate changes). All parallel mode limitations and their PRs are tracked in #16168.

plum117 and others added 5 commits October 20, 2026 20:09
…al and parallel runs

Move the status rules of the formatter's afterTest (undefined, failed,
ambiguous, the pending reason and the undefined step suffix) and the
feature stack lines into getStepResult() and getFeatureStack() in
utils.ts, so the parallel step path can use the same rules.

No change in the sequential output: the formatter tests and their
snapshots pass without edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… steps in parallel mode as in sequential mode

With parallelMode, the step path only converted the Cucumber status:

- an undefined step was reported as passed, with no error
- an ambiguous step was skipped with no reason, and
  failAmbiguousDefinitions was ignored for steps
- a failed step had the whole Cucumber message as error message, so
  reporters showed `Error: Error: ...` and internal stack lines

The parallel step path now uses getStepResult(), as the sequential
formatter does, and lists the matching step definitions of an ambiguous
step from Cucumber's support code messages.

Real-browser check (Chrome, sequential vs parallel run of the same
specs): the spec, dot, concise, JUnit and JSON reporters now give the
same step results, also with ignoreUndefinedDefinitions and
failAmbiguousDefinitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parallel mode ran the scenarios in file order and ignored
cucumberOpts.order. It now orders them with Cucumber's own
orderPickles, as Cucumber does in a sequential run: `random:<seed>`
gives the same order as a sequential run, `random` logs its seed, and
an unknown order fails before any window opens.

Real-browser check (Chrome, 2 feature files, order random:7, 1 slot):
sequential and parallel runs start the scenarios in the same order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itle in parallel mode

In parallel mode a Before/After hook had an empty title. Sequential
mode titles it with the hook name and tag expression, for example
`@beforeFails`, which the spec reporter shows for a failing hook and
the JSON reporter writes as the hook title.

The hook titles now come from the same support code messages as the
step definitions.

Real-browser check (Chrome, sequential vs parallel run of the same
specs): the spec, dot, concise, JUnit and JSON reporters give the same
result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…os in parallel mode as in sequential mode

With scenarioLevelReporter, the parallel scenario result kept its own
rules and collected an ambiguous step result before its message was
added:

- with failAmbiguousDefinitions, an ambiguous scenario failed with no
  error, and when skipped it had no pendingReason
- with ignoreUndefinedDefinitions, the scenario title had no
  ` (undefined step)` suffix
- the undefined scenario message had quotes and no scenario location

The scenario result now uses getStepResult(), as the sequential
formatter does, with the ambiguous message added first.

Real-browser check (Chrome, scenarioLevelReporter, also with
failAmbiguousDefinitions and ignoreUndefinedDefinitions): the spec,
dot, concise and JSON reporters give the same result in sequential and
parallel runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plum117
plum117 requested a review from a team as a code owner October 21, 2026 00:28
@plum117 plum117 added the PR: Bug Fix 🐛 PRs that contain bug fixes label Oct 21, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

create-wdio

npm i https://pkg.pr.new/webdriverio/webdriverio/create-wdio@16161

eslint-plugin-wdio

npm i https://pkg.pr.new/webdriverio/webdriverio/eslint-plugin-wdio@16161

wdio

npm i https://pkg.pr.new/webdriverio/webdriverio/wdio@16161

@wdio/ai-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/ai-service@16161

@wdio/allure-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/allure-reporter@16161

@wdio/appium-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/appium-service@16161

@wdio/browser-runner

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/browser-runner@16161

@wdio/cli

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/cli@16161

@wdio/concise-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/concise-reporter@16161

@wdio/config

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/config@16161

@wdio/cucumber-framework

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/cucumber-framework@16161

@wdio/cucumberjs-json-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/cucumberjs-json-reporter@16161

@wdio/display-server

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/display-server@16161

@wdio/dot-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/dot-reporter@16161

@wdio/firefox-profile-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/firefox-profile-service@16161

@wdio/globals

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/globals@16161

@wdio/jasmine-framework

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/jasmine-framework@16161

@wdio/json-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/json-reporter@16161

@wdio/junit-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/junit-reporter@16161

@wdio/lighthouse-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/lighthouse-service@16161

@wdio/local-runner

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/local-runner@16161

@wdio/logger

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/logger@16161

@wdio/mocha-framework

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/mocha-framework@16161

@wdio/protocols

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/protocols@16161

@wdio/repl

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/repl@16161

@wdio/reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/reporter@16161

@wdio/runner

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/runner@16161

@wdio/sauce-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/sauce-service@16161

@wdio/session

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/session@16161

@wdio/shared-store-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/shared-store-service@16161

@wdio/smoke-test-cjs-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/smoke-test-cjs-service@16161

@wdio/smoke-test-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/smoke-test-reporter@16161

@wdio/smoke-test-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/smoke-test-service@16161

@wdio/snapshot

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/snapshot@16161

@wdio/spec-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/spec-reporter@16161

@wdio/static-server-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/static-server-service@16161

@wdio/sumologic-reporter

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/sumologic-reporter@16161

@wdio/testingbot-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/testingbot-service@16161

@wdio/types

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/types@16161

@wdio/utils

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/utils@16161

@wdio/webdriver-mock-service

npm i https://pkg.pr.new/webdriverio/webdriverio/@wdio/webdriver-mock-service@16161

webdriver

npm i https://pkg.pr.new/webdriverio/webdriverio/webdriver@16161

webdriverio

npm i https://pkg.pr.new/webdriverio/webdriverio@16161

commit: b2bbd29

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread packages/wdio-cucumber-framework/src/cucumberFormatter.ts Outdated
plum117 and others added 3 commits October 20, 2026 20:35
…rror that shows it

getStepResult() took the feature stack as a string, so every step
result scanned the feature children, also a passing step. It now takes
a function and calls it only for the undefined and ambiguous failures,
as the formatter did before the shared rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n parallel mode

With tagsInTitle, a parallel feature suite had no tags in its title
(`Statuses in parallel` instead of `@feat: Statuses in parallel`), and
its start had no description and keyword. Reporters such as JUnit and
spec then named the feature differently than in a sequential run.

Real-browser check (Chrome, tagsInTitle, step and scenario level): the
JUnit testsuite names and the spec failure titles are the same as in a
sequential run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stop as Cucumber does

After a failFast stop, parallel mode reported each scenario that did
not run as plain skipped steps. Cucumber runs these scenarios in skip
mode: their hooks are reported as skipped, an undefined step stays
failed, and an ambiguous step keeps its reason.

They now run through Cucumber's test case runner with `skip: true`,
with no window, so the reporters get the same events as in a
sequential run. The failFast tests use a defined failing step, because
in skip mode an undefined step stays failed.

Real-browser check (Chrome, failFast, step and scenario level, 2
slots): the spec, dot, concise, JUnit and JSON reporters and the
framework events are the same as in a sequential run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread packages/wdio-cucumber-framework/src/parallel.ts Outdated
plum117 and others added 9 commits October 20, 2026 21:19
…top in the parallel run result

The scenarios that run in skip mode after a failFast stop were left out
of the run results, so a scenario that the reporters show as failed (an
undefined step) was counted as skipped in the summary and in the
failure count.

Their results now count as the other scenarios: an undefined or
ambiguous step fails the run, a skipped scenario is skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…el mode

`--spec file.feature:12` failed in parallel mode with ENOENT: the
adapter read the feature file with the `:12` suffix. With the file read,
every scenario of the file still ran, because Cucumber's line filter
compares the uri relative to the cwd and parallel pickles have an
absolute uri.

The adapter now reads the file without the line numbers and keeps them
in the specs for the line filter, which gets the pickle uri relative to
the cwd.

Real-browser check (Chrome, `--spec statuses.feature:6:22`, and 2 files
with 1 line each): the same scenarios run as in a sequential run, and
the reporters give the same result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Name what parallel mode reports like a sequential run (also with
ignoreUndefinedDefinitions, failAmbiguousDefinitions, tagsInTitle and
scenarioLevelReporter) and that specs with line numbers work, and list
the differences found in a real-browser matrix: failFast lets running
scenarios finish, ignoreUndefinedDefinitions does not make an ambiguous
or strict pending step pass the run, and Cucumber formatters write no
file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arallel mode

A pending step (also an undefined step with ignoreUndefinedDefinitions)
had `passed: true` in its parallel payload, and `passed: false` in the
sequential one. Parallel mode now uses the sequential rule: only a
passed or skipped step is passed.

Real-browser check (Chrome, ignoreUndefinedDefinitions): both pending
steps had passed=false in the sequential run and passed=true in the
parallel run before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or sequential and parallel runs

The formatter and the parallel step path each added the matching step
definitions to an ambiguous result in their own code. Both now use
withAmbiguousMessage() in utils.ts, which looks up the step only for an
ambiguous result with no message.

No change in the output: the formatter tests and snapshots, the smoke
suites and a real-browser run with failAmbiguousDefinitions give the
same result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked each option against the code and real runs:

- `timeout` defaults to 60000, `backtrace` to false (Frameworks.md said
  30000 and true)
- `strict` fails the run on a pending step; an undefined step fails it
  also without `strict` (Cucumber's shouldCauseFailure)
- `retryTagFilter` is a tag expression string, not a repeatable RegExp
- `ignoreUndefinedDefinitions` reports the step as pending with the
  ` (undefined step)` suffix and does not fail the run
- `order` explains `random:<seed>`
- a feature file with no matching tag starts no browser session (the
  runner stops before the session when the framework has no tests)
- the option is `profiles`, not `profile`; `file`, `dryRun`,
  `worldParameters` and `scenarioLevelReporter` were not documented, and
  the README did not have `parallelMode` and `maxParallelContexts`
- formatters and the message report for publishCucumberReport are not
  written with `parallelMode`
- Retry.md: Cucumber `retry` works without `scenarioLevelReporter`

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e step result rule

getStepResult() checked `willBeRetried` on a step result. Cucumber sets
this field only on the testCaseFinished message and the hook parameter,
never on a step result, so the check was always true. Retries still
work through testCaseFinished.willBeRetried, in both modes.

Docs: with ignoreUndefinedDefinitions, an ambiguous step fails a
parallel run but not a sequential one only with
failAmbiguousDefinitions: false (with true, both runs fail).

Real-browser check (Chrome, retry, retryTagFilter, scenario-level
retry): exit codes and the framework events are the same as in a
sequential run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With order: 'random', loadSources in init() and the run each picked a
seed. The sequential run logged the seed of loadSources (its own log
goes to a discarded stream), so the logged seed did not give the order
of the run again; a parallel run logged 2 seeds. init() now sets
random:<seed> and logs that seed.
…ported yet

Overlapping scenarios are nested and the passing count after a retry
can be wrong; the scenario results and the exit code are correct.
…ites

Mocha parallel mode starts all describe blocks first, so the reporter
shows each one under the one before it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR: Bug Fix 🐛 PRs that contain bug fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant