test_runner: support test order randomization · nodejs/node@d14029b · GitHub
Skip to content

Commit d14029b

Browse files
pmarchiniaduh95
pmarchini
authored and
aduh95
committed
test_runner: support test order randomization
PR-URL: #61747 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
1 parent ec8c6b9 commit d14029b

28 files changed

Lines changed: 1553 additions & 21 deletions

‎doc/api/cli.md‎

Lines changed: 35 additions & 0 deletions

‎doc/api/test.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -626,6 +626,94 @@ prevent shell expansion, which can reduce portability across systems.
626626
node --test "**/*.test.js" "**/*.spec.js"
627627
```
628628

629+
### Randomizing tests execution order
630+
631+
<!-- YAML
632+
added: REPLACEME
633+
-->
634+
635+
> Stability: 1.0 - Early development
636+
637+
The test runner can randomize execution order to help detect
638+
order-dependent tests. When enabled, the runner randomizes both discovered
639+
test files and queued tests within each file. Use `--test-randomize` to
640+
enable this mode.
641+
642+
```bash
643+
node --test --test-randomize
644+
```
645+
646+
When randomization is enabled, the test runner prints the seed used for the run
647+
as a diagnostic message:
648+
649+
```text
650+
Randomized test order seed: 12345
651+
```
652+
653+
Use `--test-random-seed=<number>` to replay the same randomized order
654+
deterministically. Supplying `--test-random-seed` also enables randomization,
655+
so `--test-randomize` is optional when a seed is provided:
656+
657+
```bash
658+
node --test --test-random-seed=12345
659+
```
660+
661+
In most test files, randomization works automatically. One important exception
662+
is when subtests are awaited one by one. In that pattern, each subtest starts
663+
only after the previous one finishes, so the runner keeps declaration order
664+
instead of randomizing it.
665+
666+
Example: this runs sequentially and is **not** randomized.
667+
668+
```mjs
669+
import test from 'node:test';
670+
671+
test('math', async (t) => {
672+
for (const name of ['adds', 'subtracts', 'multiplies']) {
673+
// Sequentially awaiting each subtest preserves declaration order.
674+
await t.test(name, async () => {});
675+
}
676+
});
677+
```
678+
679+
```cjs
680+
const test = require('node:test');
681+
682+
test('math', async (t) => {
683+
for (const name of ['adds', 'subtracts', 'multiplies']) {
684+
// Sequentially awaiting each subtest preserves declaration order.
685+
await t.test(name, async () => {});
686+
}
687+
});
688+
```
689+
690+
Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
691+
still allows randomization, because sibling tests are enqueued together.
692+
693+
Example: this remains eligible for randomization.
694+
695+
```mjs
696+
import { describe, it } from 'node:test';
697+
698+
describe('math', () => {
699+
it('adds', () => {});
700+
it('subtracts', () => {});
701+
it('multiplies', () => {});
702+
});
703+
```
704+
705+
```cjs
706+
const { describe, it } = require('node:test');
707+
708+
describe('math', () => {
709+
it('adds', () => {});
710+
it('subtracts', () => {});
711+
it('multiplies', () => {});
712+
});
713+
```
714+
715+
`--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
716+
629717
Matching files are executed as test files.
630718
More information on the test file execution can be found
631719
in the [test runner execution model][] section.
@@ -666,6 +754,10 @@ test runner functionality:
666754
* `--test-reporter` - Reporting is managed by the parent process
667755
* `--test-reporter-destination` - Output destinations are controlled by the parent
668756
* `--experimental-config-file` - Config file paths are managed by the parent
757+
* `--test-randomize` - Randomization is managed by the parent process and
758+
propagated to child processes
759+
* `--test-random-seed` - Randomization seed is managed by the parent process and
760+
propagated to child processes
669761

670762
All other Node.js options from command line arguments, environment variables,
671763
and configuration files are inherited by the child processes.
@@ -1572,6 +1664,14 @@ changes:
15721664
that specifies the index of the shard to run. This option is _required_.
15731665
* `total` {number} is a positive integer that specifies the total number
15741666
of shards to split the test files to. This option is _required_.
1667+
* `randomize` {boolean} Randomize execution order for test files and queued tests.
1668+
This option is not supported with `watch: true`.
1669+
**Default:** `false`.
1670+
* `randomSeed` {number} Seed used when randomizing execution order. If this
1671+
option is set, runs can replay the same randomized order deterministically,
1672+
and setting this option also enables randomization. The value must be an
1673+
integer between `0` and `4294967295`.
1674+
**Default:** `undefined`.
15751675
* `rerunFailuresFilePath` {string} A file path where the test runner will
15761676
store the state of the tests to allow rerunning only the failed tests on a next run.
15771677
see \[Rerunning failed tests]\[] for more information.

‎doc/node-config-schema.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,14 @@
524524
"type": "boolean",
525525
"description": "run tests with 'only' option set"
526526
},
527+
"test-random-seed": {
528+
"type": "number",
529+
"description": "seed used to randomize test execution order"
530+
},
531+
"test-randomize": {
532+
"type": "boolean",
533+
"description": "run tests in a random order"
534+
},
527535
"test-reporter": {
528536
"oneOf": [
529537
{
@@ -902,6 +910,14 @@
902910
"type": "boolean",
903911
"description": "run tests with 'only' option set"
904912
},
913+
"test-random-seed": {
914+
"type": "number",
915+
"description": "seed used to randomize test execution order"
916+
},
917+
"test-randomize": {
918+
"type": "boolean",
919+
"description": "run tests in a random order"
920+
},
905921
"test-reporter": {
906922
"oneOf": [
907923
{

‎doc/node.1‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,22 @@ Configures the type of test isolation used in the test runner.
498498
A regular expression that configures the test runner to only execute tests
499499
whose name matches the provided pattern.
500500
.
501+
.It Fl -test-random-seed
502+
Set the seed used to randomize test execution order.
503+
This applies to both test file execution order and queued tests within each file.
504+
Providing this flag enables randomization implicitly, even without
505+
\fB--test-randomize\fR.
506+
The value must be an integer between 0 and 4294967295.
507+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
508+
.
509+
.It Fl -test-randomize
510+
Randomize test execution order.
511+
This applies to both test file execution order and queued tests within each file.
512+
This can help detect tests that rely on shared state or execution order.
513+
The seed used for randomization is printed in the test summary and can be
514+
reused with \fB--test-random-seed\fR.
515+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
516+
.
501517
.It Fl -test-reporter
502518
A test reporter to use when running tests.
503519
.

‎lib/internal/test_runner/runner.js‎

Lines changed: 73 additions & 4 deletions

0 commit comments

Comments
 (0)