A test spec states what users should get, such as "Dutch users always get the new checkout", and featurevisor test checks it before any change ships. This page writes one spec of each kind, then covers checks, scenarios, matrices, target datafiles and reports.

Write each assertion's description as the promise in plain words. A spec then reads as a record of who gets what, which anyone reviewing a pull request can follow, not only the engineer who wrote it. When a change breaks a promise, the failing description says which one.

Specs can live anywhere under tests/, or sets/<set>/tests/ in a sets project. The subdirectories above are a convention, and file names need not match entity keys. End them in .spec.yml so they are easy to recognize.

In a project with environments, feature, variable and scenario specs name one with environment. The examples on this page come from a project without environments, where lint rejects the field.

Test a feature#

A feature spec names the feature, then lists assertions. Each assertion gives a context and a bucket position, and says what that user should get. This spec holds checkout to its rollout, every Dutch user and a quarter of everyone else:

feature: checkout
assertions:
  - description: Dutch users are always in
    at: 90
    context:
      country: nl
    expectedToBeEnabled: true
  - description: Others are out above 25
    at: 90
    context:
      country: de
    expectedToBeEnabled: false

at fixes the bucket position, from 0 to 100, so the result does not depend on real user IDs. Feature assertions require it. At 90, a German user is above the 25% rollout, so the second assertion expects them out.

Add expectedVariation for a feature with variations, such as pricing:

feature: pricing
assertions:
  - description: Low buckets get control
    at: 10
    expectedToBeEnabled: true
    expectedVariation: control

Add expectedVariables for its feature variables. A paying Dutch account in buyButton's treatment gets the rule's label and the paying colour:

feature: buyButton
assertions:
  - description: Paying Dutch accounts in treatment
    at: 90
    context:
      country: nl
      plan: paid
    expectedVariation: treatment
    expectedVariables:
      label: Nu kopen
      colour: '#047857'

Lint rejects expectedVariation on a feature without variations, even when the expected value is null, and suggests expectedToBeEnabled instead. Every key in expectedVariables must be declared in the feature's variablesSchema, in child assertions and matrices too. Passing a default for an undeclared variable does not make it valid.

Check the reason#

expectedEvaluations compares the evaluation details behind a result, for the flag, the variation or each of the variables. This assertion fails if Dutch users ever come in through everyone instead of their own rule, even while they still get checkout:

feature: checkout
assertions:
  - description: Dutch users match their own rule
    at: 90
    context:
      country: nl
    expectedEvaluations:
      flag:
        enabled: true
        reason: rule
        ruleKey: netherlands

Only the fields you list are compared. Evaluation reasons lists the values reason can take.

Feature assertion fields#

Children take the same evaluation fields and an optional description. Each child has its own sticky values and does not inherit its parent's.

Every spec kind names sticky values stickyFeatures and stickyVariables. Expectations differ: feature assertions use expectedEvaluations, plural, variable assertions use expectedEvaluation, and scenarios write their expectations as checks.

Run tests#

Run every spec from your project's root:

npx featurevisor test

Suppose a change raises the everyone rule of checkout to 100% by mistake. The spec above catches it before it ships:

Testing: tests/features/checkout.spec.yml (2ms)
  feature "checkout":
  ✔ Assertion #1: Dutch users are always in (1ms)
  ✘ Assertion #2: Others are out above 25 (1ms)
    => expectedToBeEnabled: expected false, received true

Each failure prints its description, the expectation that failed and a command that reruns only that case. A failed case makes test exit with a nonzero status, so a CI job stops the change. These options help while you work:

• --entity-type=feature|segment|variable|scenario runs one spec kind. It takes one value.
• --key-pattern=<pattern> filters by entity key, such as checkout, matching case.
• --only-failures hides passing results.
• --json writes a machine readable report.
• --case='<case ID>' reruns one case.

The CLI testing reference lists every option. For a CI workflow, see testing before deployment.

Lint first#

test runs assertions without linting first, so run lint before it, in CI too. Lint checks each spec against your definitions. It warns with context_value_mismatch when a context value does not fit its attribute's type or constraints, including dates, expanded matrices, scenarios and child contexts. The warning names the value, so you can tell invalid matrix values apart. Warnings do not block the run, so a spec can still test deliberately invalid input.

Test a segment#

A segment spec checks who belongs to an audience, with expectedToMatch. The scaffold comes with this one for netherlands:

segment: netherlands
assertions:
  - description: Matches users in the Netherlands
    context:
      country: nl
    expectedToMatch: true
  - description: Does not match users in Germany
    context:
      country: de
    expectedToMatch: false
  - description: Does not match when country is missing
    context: {}
    expectedToMatch: false

Segments built only from attributes need no at or environment. A segment that depends on a feature or variable, directly or through another segment, needs environment when your project configures environments. Optional at, stickyFeatures and stickyVariables then control those dependencies.

To check membership inside a test that covers several entities, use segment checks.

Test a global variable#

A variable spec names a global variable with variable. Use expectedValue for the whole value and expectedEvaluation for the evaluation fields and reasons:

variable: supportEmail
assertions:
  - description: Uses the Netherlands override
    context:
      country: nl
    expectedValue: support-[email protected]
    expectedEvaluation:
      reason: variable_override_rule
      variableOverrideKey: netherlands

Variable assertions take the same description, context, environment, target, sticky values, matrix, children, key and promotable as feature assertions, with at optional. They add defaultVariableValue, expectedValue, expectedEvaluation and checks, and need at least one of the last three or children. expectedEvaluation needs at least one field, and compares nested arrays and objects by value.

Global variables do not bucket themselves. at controls the features they depend on through conditions, including chains of them. supportEmail from Variables gives users of the new checkout its own address, so at decides whether a German user gets it. stickyFeatures fixes that result instead:

variable: supportEmail
assertions:
  - description: Inside the checkout rollout
    at: 10
    context:
      country: de
    expectedValue: checkout-[email protected]
  - description: Sticky checkout at any bucket
    at: 90
    context:
      country: de
    stickyFeatures:
      checkout:
        enabled: true
    expectedValue: checkout-[email protected]

Other feature dependencies still use at. Linting validates sticky feature keys, variations and variable keys. Use stickyVariables to fix a global variable's result.

For a variable with accumulating overrides, variableOverridePaths lists every override that matched, by key rather than by position. This spec checks the final notices and all four matched paths for a Dutch mobile user in Amsterdam:

variable: checkoutNotices
assertions:
  - context:
      country: nl
      city: amsterdam
      device: mobile
    expectedValue:
      items:
        - { id: nl, label: Amsterdam, message: Amsterdam notice }
    expectedEvaluation:
      variableOverridePaths:
        - - netherlands
        - - netherlands
          - amsterdam
        - - netherlands
          - amsterdam-message
        - - mobile-remove-base

Children test the same variable through child instances, as an application does when it adds context for one request. A child extends its parent's context and reuses its at, but has its own sticky values:

variable: supportEmail
assertions:
  - context:
      country: nl
    expectedValue: support-[email protected]
    children:
      - context:
          plan: paid
        expectedValue: [email protected]
      - stickyVariables:
          supportEmail: [email protected]
        expectedValue: [email protected]

Each child needs expectedValue, expectedEvaluation or checks. Run variable specs on their own with npx featurevisor test --entity-type=variable.

Test a composition#

Test a composed variable such as checkoutView with an ordinary variable spec. Assert the assembled object and its reason, or check single properties with paths. Sticky source values let you test a disabled flag and a null variation while the other members keep their values:

variable: checkoutView
assertions:
  - description: Disabled sources keep sibling values
    context:
      userId: 'user-9'
      country: nl
    stickyFeatures:
      checkout:
        enabled: false
      pricing:
        enabled: false
    expectedValue:
      enabled: false
      pricing: null
      label: Buy
      support: [email protected]
    expectedEvaluation:
      reason: variable_composed

Matrices and children work as they do for any global variable, and at controls the source features' buckets. A sticky value for checkoutView itself replaces the whole object without reading its members, so expect reason sticky then. If an application reads the composition through a native SDK, run the specs with that SDK's runner too.

To see which members it read and their values, ask the CLI:

npx featurevisor evaluate \
  --variable=checkoutView \
  --context='{"userId":"user-9","country":"nl"}' \
  --explain

A passing spec does not prove that every target builds. A source that is archived, or hidden from a target with expose: false, fails that target's build, so also test the target's datafile.

Repeat a test with a matrix#

A matrix repeats one assertion for every combination of the values you list. Dutch users always get checkout, so this matrix expands into six cases that all expect it:

feature: checkout
assertions:
  - matrix:
      at: [10, 90]
      plan: [free, paid, enterprise]
    description: At ${{ at }}% on ${{ plan }}
    at: ${{ at }}
    context:
      country: nl
      plan: ${{ plan }}
    expectedToBeEnabled: true

• Every matrix and axis needs at least one value, and each placeholder must name an axis.
• A value that is only a placeholder, such as at: ${{ at }}, keeps the axis value's type, including arrays and objects.
• Placeholders work in references, operands, context, at, sticky values, defaults, expectations, checks and children. Segment specs take matrices too.
• environment and target must be whole placeholders that resolve to existing names. at must resolve to a number from 0 to 100.

To see every case a matrix produces, list them. Each combination appears as its own assertion, without the matrix:

npx featurevisor list --tests --apply-matrix --json

Test a target's datafile#

By default, tests run against a datafile with every definition, built in memory. Select a target to test what one application receives instead, such as the web app:

feature: checkout
assertions:
  - target: web
    at: 90
    context:
      country: nl
    expectedToBeEnabled: true

• Set target on an assertion or a scenario. Its checks and children use it too.
• Every feature and variable read, in children and on both sides of a check, uses that exact datafile. A missing datafile or a reference it does not contain is an error, never a fallback to the whole project.
• Segment checks use the authored definitions rather than the target's datafile.

npx featurevisor test --target=web --target=mobile
npx featurevisor test --show-datafile
npx featurevisor test --set=production

• Repeat --target to run the cases for those targets. Cases without a target still run. The option filters cases and does not assign them a target, so use a matrix or separate scenarios for that.
• --show-datafile prints the datafiles the run generates.
• Without --set, a sets project tests every set, and each result and uncovered entry names its set.

Checks#

An expectation compares one result with a fixed value. A check can also compare two results with each other, such as a checkout's minimum order against the market's maximum. The test then keeps passing when both change together, and fails when they drift apart.

These examples assume two more global variables, checkoutPolicy and marketPolicy, each an object with currency, enabled and limits, alongside checkoutSettings. expectedValue compares the whole value, so it lists every property:

variable: checkoutPolicy
assertions:
  - description: Checkout policy agrees with its dependencies
    context:
      country: nl
    expectedValue:
      currency: EUR
      enabled: true
      limits:
        minimumOrder: 20
        maximumOrder: 100
    checks:
      - path: limits.minimumOrder
        operator: lessThanOrEquals
        compareTo:
          variable: marketPolicy
          path: limits.maximumOrder
      - variable: checkoutSettings
        path: currency
        operator: equals
        compareTo:
          variable: checkoutPolicy
          path: currency

• Every item in checks must pass.
• Checks can stand alone. When combined with expectedValue or expectedEvaluation, all supplied expectations must pass.
• Use a scenario when no single variable owns the relationship.

Variable checks#

Use a variable check to compare two evaluated values or compare one value with a literal:

checks:
  - variable: checkoutPolicy
    path: limits.minimumOrder
    operator: lessThanOrEquals
    compareTo:
      variable: marketPolicy
      path: limits.maximumOrder
  - feature: buyButton
    variable: label
    operator: equals
    value: Buy

• Use variable for a global variable, or feature and variable together for a feature variable. Names are literal, including dots.
• In a variable spec, a side that omits variable reads the spec's own variable. Each side decides this on its own and never borrows the other side's reference. Scenarios require explicit references.
• Comparisons take exactly one of value or compareTo. Both sides use the same instance, context, sticky values, bucket position and target.
• defaultVariableValue applies only to the owning expectation, not to checks.
• Checks use the condition operators and portable regular expression flags, without type conversion. They take neither arbitrary expressions nor deep object equality, which is what expectedValue is for.

The unary exists, notExists, isNull, notNull, empty and notEmpty operators take neither value nor compareTo. Emptiness checks use shallow string, array or object size: whitespace, [null] and { x: null } are nonempty. Null and missing paths match neither. Unsupported types cause errors when reached. Unlike notEmpty, negating empty makes null and missing paths match.

Use includesAny or notIncludesAny to compare an actual array with a candidate list. Supply a literal list with value, or resolve the list through compareTo:

checks:
  - variable: checkoutSettings
    path: paymentMethods
    operator: includesAny
    value: [card, ideal]
  - variable: checkoutSettings
    path: paymentMethods
    operator: notIncludesAny
    compareTo:
      variable: checkoutPolicy
      path: blockedPaymentMethods

Candidates must be JSON primitives: strings, numbers, booleans or null. Matching does not convert types. An empty actual or candidate array makes includesAny false and notIncludesAny true. A reached non array actual value or a malformed candidate list is a test error, including inside not. A present null actual value raises a comparison error, just like includes. Missing paths retain the usual false result. See array operators for runtime behaviour.

Portable timestamps#

Use the portable timestamp format for before and after checks. String variables can use them with or without format: date-time. An invalid timestamp fails the check that reaches it, rather than returning false as an SDK condition would.

Feature expectations#

A feature expectation supplies feature plus enabled, variation, or both:

checks:
  - feature: checkout
    enabled: true
  - feature: pricing
    variation: treatment
  - feature: pricing
    enabled: false
    variation: null

• Only supplied expectations are tested. A variation check does not imply enabled, and variation: null expects null.
• Supply at least enabled or variation. Bare { feature: checkout } and variable operand fields are invalid here. A variation check needs a feature with variations.
• The runner calls isEnabled() and getVariation() on the test instance or child. An absent feature is an error even when expecting disabled, unlike runtime conditions.

Segment checks#

Supply a segment and Boolean matches to test membership against the effective test context:

checks:
  - segment: payingAccounts
    matches: true
  - segment: netherlands
    matches: false

• Segment checks accept only segment, matches, key and description. Use them in variable assertions, scenarios, groups, matrices or children.
• They evaluate authored definitions from the active project or set. Attribute leaves use the assertion context, not a target's build context. Feature and variable leaves inherit the enclosing assertion's environment, at, sticky features and sticky variables.
• Unknown segments are errors even when expecting no match. Catalog's Tests tab associates only direct segment checks.

Groups#

A list item can contain exactly one nonempty and, or or not group. Groups may nest and combine all three leaf forms:

checks:
  - or:
      - feature: pricing
        variation: treatment
      - variable: checkoutPolicy
        path: channel
        operator: equals
        value: preview
  - not:
      - variable: checkoutPolicy
        path: retired
        operator: equals
        value: true

Groups run in order and stop once their result is known. not negates its children's implied AND.

All branches are validated before evaluation, including skipped ones. Evaluation errors fail the case even inside or or not.

Paths, null and missing values#

Omit path or use [] for the complete getter result. Paths follow conditions traversal. A dotted path splits into components and converts canonical nonnegative integers to array indices. An array path preserves component types:

checks:
  - variable: checkoutPolicy
    path: items.0.limit
    operator: equals
    value: 4
  - variable: checkoutPolicy
    path: [items, "0", limit]
    operator: equals
    value: 4
  - variable: checkoutPolicy
    path: [items, length]
    operator: equals
    value: 1
  - variable: checkoutPolicy
    path: [items, "00"]
    operator: notExists

Use object keys and array indices, or length for array size. See path traversal rules for valid components.

null satisfies exists and isNull. Missing paths satisfy notExists. notNull requires a present nonnull value. Missing variable getters return null. Other operators return false for missing paths, including negative operators. not can invert that false result, but cannot invert resolution or type errors.

Name checks and groups#

Name a check to make failures and Catalog links easier to identify:

checks:
  - key: valid-currency
    description: Checkout uses an allowed currency
    variable: checkoutPolicy
    path: currency
    operator: in
    value: [EUR, CHF]

• Leaves and groups accept optional key and description without changing evaluation.
• Keys must be nonempty and unique among siblings. Descriptions are strings.
• Named and unnamed siblings can mix.

Scenarios#

A scenario is a spec made only of checks, for a rule that no single feature or variable owns. It runs only in tests and adds nothing to datafiles or SDKs. This one holds the web checkout to two promises for Dutch users: the policy uses the checkout's currency, and its minimum order stays within the market's maximum:

description: Web checkout uses the market currency and valid limits
target: web
context:
  country: nl
checks:
  - variable: checkoutPolicy
    path: currency
    operator: equals
    compareTo:
      variable: checkoutSettings
      path: currency
  - variable: checkoutPolicy
    path: limits.minimumOrder
    operator: lessThanOrEquals
    compareTo:
      variable: marketPolicy
      path: limits.maximumOrder

• A scenario has no feature, segment or variable field, and needs at least one check. Its file path identifies it.
• Add context, at, stickyFeatures or stickyVariables to set what it evaluates with.
• Use environment, target or matrix to choose configurations, and description or promotable to describe and manage it.
• Add children to test child instances. Each child needs its own checks, and may add context, sticky values and a description.

Explicit references#

Every variable reference in a scenario, compareTo included, names its variable, because there is no owner to fall back to. A feature variable also names its feature. For deep equality, use expectedValue in a global variable spec.

Sticky values and children#

The root's context, sticky values and at apply to every check. As in variable specs, at controls feature dependencies, not global variable bucketing. Children extend the context and keep at, but need their own sticky values:

at: 75
context:
  country: nl
checks:
  - variable: checkoutPolicy
    path: currency
    operator: equals
    value: EUR
children:
  - context:
      city: amsterdam
    stickyFeatures:
      pricing:
        enabled: true
        variation: treatment
    checks:
      - feature: pricing
        variation: treatment

Repeat a scenario#

A scenario declares its matrix at the root, so every check repeats for each combination. This one runs the currency check for both countries in both applications:

matrix:
  country: [nl, de]
  target: [web, mobile]
description: ${{ target }} checkout in ${{ country }}
target: ${{ target }}
context:
  country: ${{ country }}
checks:
  - variable: checkoutPolicy
    path: currency
    operator: in
    value: [EUR, CHF]

Run and inspect scenarios#

npx featurevisor test --entity-type=scenario
npx featurevisor list --scenarios --json --pretty

list --scenarios shows each scenario as written, with its matrix unexpanded. list --tests --json lists scenarios alongside entity assertions, and --apply-matrix expands them. In test results, each scenario case counts as one assertion, children included. It is named by its description, prefixed with Case N of M when a matrix produced it, which does not change its identity in reports.

Run from npm scripts#

Call featurevisor test from an npm script, and pass further options after --:

{
  "scripts": {
    "test": "featurevisor test"
  }
}

npm test -- --key-pattern=checkout

Reports, reruns and coverage#

JSON report#

npx featurevisor test --json
npx featurevisor test --json --pretty

The report is for CI and for tools such as Catalog. Version 1 has schemaVersion, runner, passed, summary, results and errors. The summary counts specs and assertions in total, and entities and scenarios separately. Each failure keeps whatever path, values and operand evidence it has. Durations are optional.

This excerpt is the report for the failing checkout assertion above:

{
  "schemaVersion": 1,
  "runner": "javascript",
  "passed": false,
  "results": [
    {
      "source": "tests/features/checkout.spec.yml",
      "cases": [
        {
          "id": "[null,\"feature\",\"features.checkout\",1,null]",
          "description": "Assertion #2: Others are out above 25",
          "passed": false,
          "errors": [
            {
              "path": ["assertions", 1, "expectedToBeEnabled"],
              "expected": false,
              "actual": true
            }
          ],
          "rerun": {
            "command": "npx featurevisor",
            "args": [
              "test",
              "--case=[null,\"feature\",\"features.checkout\",1,null]"
            ]
          }
        }
      ]
    }
  ]
}

A missing value appears as { "$type": "undefined" }, so it is never confused with null.

Each spec's key keeps its directories and drops only the final .spec, so tests/features/checkout.spec.yml becomes features.checkout with the default namespace character. source keeps the file path. checkout.yml and checkout.spec.yml cannot share a directory, because their keys would collide.

JavaScript reports include an input fingerprint under provenance when preparing it succeeds, except with --inflate. Catalog uses it to label results current or stale. If the fingerprint cannot be prepared, the CLI warns and continues without it. Failures in inputs the tests need still fail the run.

Import report types from @featurevisor/core:

import type {
  CoverageSelection,
  ReportCase,
  ReportError,
  ReportSpec,
  TestReport,
} from '@featurevisor/core'

Automation must check the exit status as well as the report. Failures return nonzero, including failures before a JSON report can be written.

Exact case reruns#

Every failed case displays a command with an opaque --case ID:

npx featurevisor test --case='<case ID>'

Case IDs are assigned before filtering, and each selects one expanded case. Entity, key, assertion, set and target filters still apply, and an ID that matches nothing is an error. The JSON report gives rerun as raw arguments, and the human output quotes them for the shell.

A native runner's rerun calls that SDK's executable. Java supplies argumentEncoding: "maven-exec" and wraps the arguments in -Dexec.args. When the run named a project directory, the rerun names it too: Node.js accepts --root-directory-path, and reports may use its --rootDirectoryPath alias. Native runners use --projectDirectoryPath. Otherwise, rerun from the same directory.

Definitions without associated specs#

npx featurevisor test --uncovered
npx featurevisor test --json --uncovered

--uncovered lists features and global variables that no spec mentions, across the whole suite whatever the filters. References from checks and children count, as in list --features --without-tests and list --variables --without-tests. Use it to find a feature nobody wrote a spec for. It does not measure which code paths the specs reach.

Observed selections#

npx featurevisor test --coverage
npx featurevisor test --json --coverage

--coverage records which keyed rules and overrides the tests selected, without evaluating anything extra. Use it to spot a rule no spec reaches, such as a market added without a test. The report uses mode: "observed-selections", with sorted, deduplicated records naming the set, environment, target, owner and ruleKey or overridePath. Accumulating paths appear separately.

An empty list means no selection was observed. Coverage does not measure branch coverage, capture every nested evaluation or prove behaviour for contexts you did not test.

Native SDK runners#

Each native SDK ships a runner that executes these same specs through that SDK, so an application in Go or Python is tested against the evaluation it really uses. The runners use @featurevisor/cli 4.0.0 or later to parse the project and build datafiles. The SDK runtimes themselves have no Node.js dependency.

Commands are documented for Go, Python, PHP, Ruby, Java, Swift, Rust and Elixir. Contributors can read the execution protocol notes.

Promotable tests and scenarios#

In sets projects, promotable: false protects an existing destination spec from promotion. A missing destination spec is still created, and keeps the field. Specs are matched by file path.

The examples below come from a project with sets and no environments, so a set named production is not an environment. Add environment to their assertions if your project also configures environments.

feature: checkout
promotable: false
assertions:
  - at: 50
    expectedToBeEnabled: false

To protect one assertion, give every assertion a unique stable key, then mark the protected one:

feature: checkout
assertions:
  - key: production-rollout
    promotable: false
    at: 50
    expectedToBeEnabled: false
  - key: general-rollout
    at: 10
    expectedToBeEnabled: true

Protected source assertions are omitted. Protected destination assertions survive matching source keys. If any assertion has a key, all need unique keys. Assertion protection requires keys in both source and destination specs. Unkeyed arrays promote as a whole. Matrix cases and children cannot be protected separately. Keys also label Catalog permalinks, with expanded labels such as production-rollout.1.

Set promotable: false on a scenario root to protect the whole destination scenario, including matrix cases and children. This also uses sets without environments. Add environment at the scenario root only if the project configures environments:

promotable: false
checks:
  - variable: checkoutPolicy
    path: enabled
    operator: equals
    value: true

Related#

• Global variables
• Condition operators
• Catalog, which displays specs and saved results
• Testing before deployment