{{ message }}
Repository navigation
Conversation
…al and parallel runs Move the status rules of the formatter's afterTest (undefined, failed, ambiguous, the pending reason and the undefined step suffix) and the feature stack lines into getStepResult() and getFeatureStack() in utils.ts, so the parallel step path can use the same rules. No change in the sequential output: the formatter tests and their snapshots pass without edits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… steps in parallel mode as in sequential mode With parallelMode, the step path only converted the Cucumber status: - an undefined step was reported as passed, with no error - an ambiguous step was skipped with no reason, and failAmbiguousDefinitions was ignored for steps - a failed step had the whole Cucumber message as error message, so reporters showed `Error: Error: ...` and internal stack lines The parallel step path now uses getStepResult(), as the sequential formatter does, and lists the matching step definitions of an ambiguous step from Cucumber's support code messages. Real-browser check (Chrome, sequential vs parallel run of the same specs): the spec, dot, concise, JUnit and JSON reporters now give the same step results, also with ignoreUndefinedDefinitions and failAmbiguousDefinitions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parallel mode ran the scenarios in file order and ignored cucumberOpts.order. It now orders them with Cucumber's own orderPickles, as Cucumber does in a sequential run: `random:<seed>` gives the same order as a sequential run, `random` logs its seed, and an unknown order fails before any window opens. Real-browser check (Chrome, 2 feature files, order random:7, 1 slot): sequential and parallel runs start the scenarios in the same order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itle in parallel mode In parallel mode a Before/After hook had an empty title. Sequential mode titles it with the hook name and tag expression, for example `@beforeFails`, which the spec reporter shows for a failing hook and the JSON reporter writes as the hook title. The hook titles now come from the same support code messages as the step definitions. Real-browser check (Chrome, sequential vs parallel run of the same specs): the spec, dot, concise, JUnit and JSON reporters give the same result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…os in parallel mode as in sequential mode With scenarioLevelReporter, the parallel scenario result kept its own rules and collected an ambiguous step result before its message was added: - with failAmbiguousDefinitions, an ambiguous scenario failed with no error, and when skipped it had no pendingReason - with ignoreUndefinedDefinitions, the scenario title had no ` (undefined step)` suffix - the undefined scenario message had quotes and no scenario location The scenario result now uses getStepResult(), as the sequential formatter does, with the ambiguous message added first. Real-browser check (Chrome, scenarioLevelReporter, also with failAmbiguousDefinitions and ignoreUndefinedDefinitions): the spec, dot, concise and JSON reporters give the same result in sequential and parallel runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create-wdio
eslint-plugin-wdio
wdio
@wdio/ai-service
@wdio/allure-reporter
@wdio/appium-service
@wdio/browser-runner
@wdio/cli
@wdio/concise-reporter
@wdio/config
@wdio/cucumber-framework
@wdio/cucumberjs-json-reporter
@wdio/display-server
@wdio/dot-reporter
@wdio/firefox-profile-service
@wdio/globals
@wdio/jasmine-framework
@wdio/json-reporter
@wdio/junit-reporter
@wdio/lighthouse-service
@wdio/local-runner
@wdio/logger
@wdio/mocha-framework
@wdio/protocols
@wdio/repl
@wdio/reporter
@wdio/runner
@wdio/sauce-service
@wdio/session
@wdio/shared-store-service
@wdio/smoke-test-cjs-service
@wdio/smoke-test-reporter
@wdio/smoke-test-service
@wdio/snapshot
@wdio/spec-reporter
@wdio/static-server-service
@wdio/sumologic-reporter
@wdio/testingbot-service
@wdio/types
@wdio/utils
@wdio/webdriver-mock-service
webdriver
webdriverio
commit: |
Contributor
…rror that shows it getStepResult() took the feature stack as a string, so every step result scanned the feature children, also a passing step. It now takes a function and calls it only for the undefined and ambiguous failures, as the formatter did before the shared rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n parallel mode With tagsInTitle, a parallel feature suite had no tags in its title (`Statuses in parallel` instead of `@feat: Statuses in parallel`), and its start had no description and keyword. Reporters such as JUnit and spec then named the feature differently than in a sequential run. Real-browser check (Chrome, tagsInTitle, step and scenario level): the JUnit testsuite names and the spec failure titles are the same as in a sequential run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stop as Cucumber does After a failFast stop, parallel mode reported each scenario that did not run as plain skipped steps. Cucumber runs these scenarios in skip mode: their hooks are reported as skipped, an undefined step stays failed, and an ambiguous step keeps its reason. They now run through Cucumber's test case runner with `skip: true`, with no window, so the reporters get the same events as in a sequential run. The failFast tests use a defined failing step, because in skip mode an undefined step stays failed. Real-browser check (Chrome, failFast, step and scenario level, 2 slots): the spec, dot, concise, JUnit and JSON reporters and the framework events are the same as in a sequential run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…top in the parallel run result The scenarios that run in skip mode after a failFast stop were left out of the run results, so a scenario that the reporters show as failed (an undefined step) was counted as skipped in the summary and in the failure count. Their results now count as the other scenarios: an undefined or ambiguous step fails the run, a skipped scenario is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…el mode `--spec file.feature:12` failed in parallel mode with ENOENT: the adapter read the feature file with the `:12` suffix. With the file read, every scenario of the file still ran, because Cucumber's line filter compares the uri relative to the cwd and parallel pickles have an absolute uri. The adapter now reads the file without the line numbers and keeps them in the specs for the line filter, which gets the pickle uri relative to the cwd. Real-browser check (Chrome, `--spec statuses.feature:6:22`, and 2 files with 1 line each): the same scenarios run as in a sequential run, and the reporters give the same result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Name what parallel mode reports like a sequential run (also with ignoreUndefinedDefinitions, failAmbiguousDefinitions, tagsInTitle and scenarioLevelReporter) and that specs with line numbers work, and list the differences found in a real-browser matrix: failFast lets running scenarios finish, ignoreUndefinedDefinitions does not make an ambiguous or strict pending step pass the run, and Cucumber formatters write no file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arallel mode A pending step (also an undefined step with ignoreUndefinedDefinitions) had `passed: true` in its parallel payload, and `passed: false` in the sequential one. Parallel mode now uses the sequential rule: only a passed or skipped step is passed. Real-browser check (Chrome, ignoreUndefinedDefinitions): both pending steps had passed=false in the sequential run and passed=true in the parallel run before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or sequential and parallel runs The formatter and the parallel step path each added the matching step definitions to an ambiguous result in their own code. Both now use withAmbiguousMessage() in utils.ts, which looks up the step only for an ambiguous result with no message. No change in the output: the formatter tests and snapshots, the smoke suites and a real-browser run with failAmbiguousDefinitions give the same result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked each option against the code and real runs: - `timeout` defaults to 60000, `backtrace` to false (Frameworks.md said 30000 and true) - `strict` fails the run on a pending step; an undefined step fails it also without `strict` (Cucumber's shouldCauseFailure) - `retryTagFilter` is a tag expression string, not a repeatable RegExp - `ignoreUndefinedDefinitions` reports the step as pending with the ` (undefined step)` suffix and does not fail the run - `order` explains `random:<seed>` - a feature file with no matching tag starts no browser session (the runner stops before the session when the framework has no tests) - the option is `profiles`, not `profile`; `file`, `dryRun`, `worldParameters` and `scenarioLevelReporter` were not documented, and the README did not have `parallelMode` and `maxParallelContexts` - formatters and the message report for publishCucumberReport are not written with `parallelMode` - Retry.md: Cucumber `retry` works without `scenarioLevelReporter` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e step result rule getStepResult() checked `willBeRetried` on a step result. Cucumber sets this field only on the testCaseFinished message and the hook parameter, never on a step result, so the check was always true. Retries still work through testCaseFinished.willBeRetried, in both modes. Docs: with ignoreUndefinedDefinitions, an ambiguous step fails a parallel run but not a sequential one only with failAmbiguousDefinitions: false (with true, both runs fail). Real-browser check (Chrome, retry, retryTagFilter, scenario-level retry): exit codes and the framework events are the same as in a sequential run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With order: 'random', loadSources in init() and the run each picked a seed. The sequential run logged the seed of loadSources (its own log goes to a discarded stream), so the logged seed did not give the order of the run again; a parallel run logged 2 seeds. init() now sets random:<seed> and logs that seed.
…ported yet Overlapping scenarios are nested and the passing count after a retry can be wrong; the scenario results and the exit code are correct.
…ites Mocha parallel mode starts all describe blocks first, so the reporter shows each one under the one before it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
With
cucumberOpts.parallelMode, the parallel step path only converted the Cucumber status. The sequential formatter has more rules. All reporters got different step results in a parallel run:failedwithStep "…" is not defined. …, orpending+(undefined step)withignoreUndefinedDefinitionsfailAmbiguousDefinitionsis ignored for stepspendingReason, or failed withfailAmbiguousDefinitionsError: Error: …and internal stack linescucumberOpts.orderis ignoredorderPickles:random:<seed>gives the sequential order, an unknown order fails before any window opensBefore/Afterhook has an empty title@beforeFailstagsInTitle, a feature title has no tags; the feature start has no description and keyword@feat: Statuses in parallel, with description and keywordfailFaststop, the scenarios that did not run are plain skipped steps, and are left out of the run resultSKIPPED, an undefined step stays failed (and counts as a failure), an ambiguous step keeps its reasonorder: 'random', the run uses another seed than the logged one (sequential), or logs 2 seeds (parallel)init()and logged once;random:<logged seed>gives the same order again, in both modespassed: truein its parallel payload (sequential:false)passed: false, as sequential--spec file.feature:12fails withENOENT(the file is read with the:12); with the file read, all its scenarios runG1 and G2 also apply with
scenarioLevelReporter: an ambiguous scenario failed with no error or was skipped with no reason, and an ignored undefined scenario had no(undefined step)suffix. The scenario result now uses the same rule.Docs:
ParallelTests.mdlists what parallel mode reports like a sequential run and the real differences; it and the spec reporter README now say that@wdio/spec-reporterdoes not support Cucumber parallel mode yet (as Allure and cucumberjs-json). ThecucumberOptsdocs (README,Frameworks.md, JSDoc) are corrected against the code:timeout/backtracedefaults,strict,retryTagFilter,ignoreUndefinedDefinitions,order,profiles, the tags session note, and the missingfile,dryRun,worldParameters,scenarioLevelReporter,parallelMode,maxParallelContexts.Retry.md: Cucumberretryworks withoutscenarioLevelReporter.Evidence
Full matrix (headless Chrome, BiDi): 49 cases, each a sequential run and a parallel run of the same specs. Exit codes and the framework events of every scenario are the same, except the documented differences below. Cases:
contexts/userContexts, 1/4/8 slots, fallback,ignoreUndefinedDefinitions×failAmbiguousDefinitions×strict(8),tagsInTitle,retry,retryTagFilter,order(random:7,random, rejected values),dryRun,failFast,tags,name, French, failedBeforeAll, line numbers, same file names,@skip(...), Rule/Background/Outline with tagged Examples/doc string/table/empty scenario/*/But, failingAfter, step timeout,BeforeStep/AfterStep, and the scenario-level variants.Documented differences (
ParallelTests.md):failFastlets the scenarios that already run finish; withignoreUndefinedDefinitions: true, an ambiguous step (withfailAmbiguousDefinitions: false) or astrictpending step fails the run (a sequential run then passes); Cucumber formatters write no file.Real browser (headless Chrome, BiDi): the same 3 feature files run sequential, then with
parallelMode: 'contexts'(4 slots). Each reporter's output is compared between the 2 runs.S undefined✓ passed; 2 failing instead of 3;Error: Error: fails on purpose+onTestStepFinishedframes; failing hook shown as✖Finstead of 3F<failure>; failure message with the stackpassed; hook titles''failFast, 2 slots)Also the same as sequential with
ignoreUndefinedDefinitions: true, withfailAmbiguousDefinitions: true, withscenarioLevelReporter: true(alone and with both options), withtagsInTitle(step and scenario level), withretry,dryRun,tags,name,userContexts, 1 and 8 slots, and withorder: 'random:7'(scenario start orderD tables, C, X, Y, D dropdown, A, Bin both runs). WithscenarioLevelReporter, JUnit puts the session-endDELETE /sessioncommand log under the last finished scenario, which is a different scenario in a parallel run; that is the reporter's rule for commands with no browsing context, not a step result.@wdio/cucumber-framework(new:getStepResult,getFeatureStack, parallel step and scenario results, hook titles, order, the feature stack only for errors, feature titles, failFast skip mode, 1 random seed); the 2 older failFast tests now use a defined failing step, because skip mode keeps an undefined step failed; each new test failed before its fix.random:<logged seed>gives the same scenario order (sequentialrandom:64756, parallelrandom:550536), and 1 seed gives the same order in both modes.Merge Danger
Door: two-way
Blast Radius: Cucumber parallel mode only (with
failFast, the run result does not change: the run already failed). Sequential runs use the same rules as before, moved intogetStepResult(); withorder: 'random', a sequential run now logs the seed that it uses.A parallel run with an undefined step now reports that step as failed. The run already failed before (the run result counted it), so no passing run starts to fail.
Out of scope:
@wdio/allure-reporter,@wdio/spec-reporterand@wdio/cucumberjs-json-reporterstill mix overlapping Cucumber scenarios in parallel mode (their own state, separate changes). All parallel mode limitations and their PRs are tracked in #16168.