test_runner: extend tag filter with boolean expression DSL · nodejs/node@e3fda69 · GitHub
Skip to content

Commit e3fda69

Browse files
atlowChemiaduh95
authored andcommitted
test_runner: extend tag filter with boolean expression DSL
Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il> PR-URL: #63054 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
1 parent 575260d commit e3fda69

11 files changed

Lines changed: 920 additions & 97 deletions

doc/api/cli.md

Lines changed: 14 additions & 8 deletions

doc/api/test.md

Lines changed: 76 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -489,8 +489,8 @@ added: v26.2.0
489489
490490
Tags annotate tests and suites with arbitrary string labels. The
491491
[`--experimental-test-tag-filter`][] CLI flag (or the `testTagFilters`
492-
option on [`run()`][]) selects tests whose tag set contains every
493-
provided filter value.
492+
option on [`run()`][]) selects tests by a boolean expression over those
493+
labels.
494494

495495
Tags are an alternative to encoding metadata into test names. They are
496496
useful for cross-cutting axes such as subsystem, speed bucket, flakiness,
@@ -523,37 +523,89 @@ describe('database', { tags: ['db'] }, () => {
523523
});
524524
```
525525

526-
Tag values must be non-empty strings. Tags are matched case-insensitively;
527-
the canonical form is lowercase. Duplicates within a single `tags` array
528-
are collapsed on the lowercased form, preserving the first-seen
529-
declaration order.
526+
Tag values must be non-empty strings that contain no whitespace, no
527+
operator characters (`& | ! ( ) *`), and are not the reserved words
528+
`'and'`, `'or'`, or `'not'` in any casing. Tags are matched
529+
case-insensitively; the canonical form is lowercase. Duplicates within a
530+
single `tags` array are collapsed on the lowercased form, preserving the
531+
first-seen declaration order.
530532

531533
Hooks (`before`, `after`, `beforeEach`, `afterEach`) do not declare their
532534
own tags. They run as part of their owning suite, which carries the
533535
suite's tags.
534536

535-
### Filtering by tag
537+
### Filtering syntax
536538

537-
Each [`--experimental-test-tag-filter`][] value is a literal tag name. A
538-
test runs only when its tag set contains that name. The flag may be
539-
specified more than once; tests must match **every** filter to run. The
540-
same applies to the `testTagFilters` array on [`run()`][]. Filters are
541-
case-insensitive and AND'd with [`--test-name-pattern`][],
542-
[`--test-skip-pattern`][], and `.only` filtering.
539+
The filter expression supports:
543540

544-
Untagged tests are excluded under any non-empty filter, since the filter
545-
requires the tag to be present.
541+
* Identifiers—any non-whitespace, non-operator characters. A literal
542+
identifier matches a tag of the same value (case-insensitive).
543+
* `*` wildcards inside an identifier match any sequence of characters.
544+
A bare `*` matches any tagged test.
545+
* Boolean operators with two equivalent forms:
546+
* `and` / `&&`
547+
* `or` / `||`
548+
* `not` / `!`
549+
* Parentheses for grouping.
546550

547-
### Reading tags from inside a test
551+
The word forms (`and`, `or`, `not`) require whitespace separation; the
552+
punctuation forms do not.
553+
554+
#### Operator precedence
555+
556+
The expression is evaluated with the standard precedence
557+
`not > and > or`. Binary operators are left-associative.
558+
559+
| Expression | Equivalent grouping |
560+
| -------------- | ------------------- |
561+
| `a or b and c` | `a or (b and c)` |
562+
| `not a and b` | `(not a) and b` |
563+
564+
Use parentheses to override:
565+
566+
| Expression | Selects |
567+
| ------------------------------ | ------------------------------------------ |
568+
| `(unit or smoke) and not slow` | unit-or-smoke tests that are not also slow |
569+
| `db && !flaky` | db tests that are not flaky |
570+
| `*` | every tagged test |
571+
572+
#### Untagged tests
573+
574+
Untagged tests behave as if they have an empty tag set. As a result:
575+
576+
| Filter expression | Untagged test | Why |
577+
| ------------------------ | ------------- | ------------------------------------------------ |
578+
| `db` | excluded | Positive match against an empty tag set is false |
579+
| `*` | excluded | The bare wildcard requires at least one tag |
580+
| `db or unit` | excluded | Both branches are false against an empty tag set |
581+
| `not flaky` | included | Negation against an empty tag set is true |
582+
| `not flaky and not slow` | included | Both negations are true against an empty tag set |
583+
| `db or not flaky` | included | The negated branch is true |
584+
585+
For example, `--experimental-test-tag-filter='not flaky'` runs every test
586+
that is not tagged `flaky`, including all untagged tests.
587+
588+
#### Composing multiple filters
589+
590+
[`--experimental-test-tag-filter`][] may be specified more than once on the
591+
command line. Multiple expressions compose by AND—a test must satisfy
592+
every expression to run. The same applies to passing an array to
593+
`testTagFilters` on [`run()`][]. The tag filter is also AND'd with
594+
[`--test-name-pattern`][], [`--test-skip-pattern`][], and `.only`
595+
filtering.
596+
597+
#### Reading tags from inside a test
548598

549599
The [`TestContext`][] object exposes the test's tags as a frozen array
550600
through [`context.tags`][], so tests can branch on their own metadata.
551601

552-
### Errors
602+
#### Errors
553603

554604
A tag value that violates the validation rules above throws
555605
`ERR_INVALID_ARG_VALUE` at the registration site, before any test runs.
556-
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`.
606+
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`. A malformed
607+
filter expression on the CLI causes the test runner to exit with a
608+
non-zero status before running any test files.
557609

558610
## Extraneous asynchronous activity
559611

@@ -826,7 +878,7 @@ test runner functionality:
826878

827879
* `--test` - Prevented to avoid recursive test execution
828880
* `--experimental-test-coverage` - Managed by the test runner
829-
* `--experimental-test-tag-filter` - Filter values are validated by the parent
881+
* `--experimental-test-tag-filter` - Filter expressions are validated by the parent
830882
process and re-emitted to child processes
831883
* `--watch` - Watch mode is handled at the parent level
832884
* `--experimental-default-config-file` - Config file loading is handled by the parent
@@ -1740,10 +1792,11 @@ changes:
17401792
For each test that is executed, any corresponding test hooks, such as
17411793
`beforeEach()`, are also run.
17421794
**Default:** `undefined`.
1743-
* `testTagFilters` {string|string\[]} A tag name, or an array of tag names,
1744-
used to filter tests by their declared tags. Tests must contain every
1745-
listed tag to run. Equivalent to passing [`--experimental-test-tag-filter`][]
1746-
on the command line. See [Test tags][]. **Default:** `undefined`.
1795+
* `testTagFilters` {string|string\[]} A boolean expression, or an array of
1796+
boolean expressions, used to filter tests by their declared tags.
1797+
Multiple expressions compose by AND. Equivalent to passing
1798+
[`--experimental-test-tag-filter`][] on the command line. See
1799+
[Test tags][]. **Default:** `undefined`.
17471800
* `timeout` {number} A number of milliseconds the test execution will
17481801
fail after.
17491802
If unspecified, subtests inherit this value from their parent.

doc/node.1

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -810,14 +810,18 @@ collecting code coverage from tests for more details.
810810
Enable module mocking in the test runner.
811811
This feature requires \fB--allow-worker\fR if used with the Permission Model.
812812
.
813-
.It Fl -experimental-test-tag-filter Ns = Ns Ar <tag>
814-
Run only tests whose tag set contains \fB<tag>\fR. Tests declare tags via the
815-
\fBtags\fR option on \fBtest()\fR, \fBit()\fR, \fBsuite()\fR, or \fBdescribe()\fR; tags
816-
inherit from suites to nested tests by union. Filtering is
817-
case-insensitive.
818-
The flag may be specified more than once; tests must contain \fBevery\fR
819-
filter value to run. See Test tags for details on declaring and
820-
inheriting tags.
813+
.It Fl -experimental-test-tag-filter Ns = Ns Ar '<expr>'
814+
Run only tests that match the provided boolean tag-filter expression. Tests
815+
declare tags via the \fBtags\fR option on \fBtest()\fR, \fBit()\fR, \fBsuite()\fR, or
816+
\fBdescribe()\fR. Tags inherit from suites to nested tests by union.
817+
The expression supports boolean operators (\fBand\fR/\fB&&\fR, \fBor\fR/\fB||\fR,
818+
\fBnot\fR/\fB!\fR), parentheses for grouping, and \fB*\fR wildcards inside identifiers.
819+
Standard precedence applies: \fBnot\fR binds tighter than \fBand\fR, which binds
820+
tighter than \fBor\fR. See Test tags for the full grammar and behavior.
821+
The flag may be specified more than once; multiple expressions are combined
822+
with AND, so a test must satisfy every expression to run.
823+
A malformed expression causes the test runner to exit with a non-zero status
824+
before running any tests.
821825
.
822826
.It Fl -experimental-vfs
823827
Enable the experimental \fBnode:vfs\fR module.

lib/internal/test_runner/runner.js

Lines changed: 27 additions & 8 deletions

0 commit comments

Comments
 (0)