feat: add Appendix F, provider conformance (TCK) by aepfli · Pull Request #439 · open-feature/spec · GitHub
Skip to content

feat: add Appendix F, provider conformance (TCK) - #439

Draft
aepfli wants to merge 1 commit into
mainfrom
feat/provider-tck-appendix-doc
Draft

aepfli wants to merge 1 commit into
mainfrom
feat/provider-tck-appendix-doc

Conversation

@aepfli

@aepfli aepfli commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Part of #417. The assets this describes are on main: #423 merged on 14 October. This is the prose half — the normative description of what is already in the tree.

These were one pull request until the split, which existed to let the assets land while this prose was still in review — four language implementations pin them, and every rebase of the branch invalidated those pins. That worked: the assets are in, the pins are stable, and nothing here blocks an implementation any more.

What

Appendix F: Provider Conformance (TCK) — the normative description of the conformance assets now on main (#423). What a TCK implementation must do, what each capability asserts, the control-API contract, the extension point, and the run-integrity checks.

Marked experimental and non-normative.

How the requirements are expressed

As normative blockquotes (58 of them), not numbered requirements — following Appendix A, since specification.json is built from the numbered sections and no appendix contributes to it.

Whether any of this should become numbered normative sections is a TSC decision, recorded as an open question rather than assumed. That is the main thing I would like a view on.

Design decisions worth reviewing

Each is argued in the appendix; this is the index.

  • Capabilities. Optional behaviour is tagged; a provider declares what it supports and the rest are skipped with a reason, never passed. Tags with no scenarios yet are reserved and must not be declarable.
  • @lifecycle and @events are separate, because a provider can initialise against its backend without emitting events, and vice versa.
  • @numeric-coercion is borrowed, not specified. The rule comes from a flagd ADR, not from the specification. An earlier draft claimed the opposite and called withholding the tag "an admission of a known bug" — wrong, and corrected. Gap: #430.
  • @string-typing and @fully-typed-values are two tags because one hid a bug. Three providers over one Flagsmith backend: two report TYPE_MISMATCH for a boolean and an integer asked as strings, one stringifies them, and all three stringify the float and the structure. The latter is the backend's shape, the former a provider defect; one tag reported the defect as a permitted absence. Gap: #433.
  • No container restarts. Outages are produced through the control API, because orchestrators reassign host ports on restart and every provider pointed at the old one then fails looking flaky.
  • In-process control is a narrow carve-out for providers with no backend. A provider with a backend must use the control API, or the suite passes while proving nothing.
  • An extension point is required, not optional — raised in review on the original PR. An extension may add questions but never replace one: it cannot occupy a canonical path, cannot satisfy a canonical scenario, and a run that skipped part of the canonical set must fail. That last rule was found by accident — a selector matching one scenario name produced a green suite and a well-formed report covering one scenario.
  • Run-integrity checks. Three failures are invisible in results and must fail the run; the one that matters is a scenario carrying a tag the implementation does not know, which all four reference implementations ignored. An unknown tag gates nothing, so its scenarios stay mandatory and a suite that has not learned a capability silently keeps demanding the old behaviour. Two further rules are about the checks themselves: a check that cannot be performed must fail rather than skip, and the revision check must be in force where the scenarios execute. Both came from one adoption producing entirely plausible numbers against the previous revision's scenarios.

One home for the vocabulary

assets/provider-tck/README.md carries the capability table on main, so the tags are defined while this is in review. This PR moves it here and leaves a pointer behind — the appendix owns it, and two copies would be able to disagree.

Open questions

  • Evaluation context passthrough — the targeting key is covered, no other attribute is; needs an echo operation on the control API.
  • Accessor width — @large-integers covers up to 2^53 − 1; 32- versus 64-bit accessors are not modelled. Part of #430.
  • Finer-grained flag manipulation would need new control-API endpoints.
  • Flag metadata has one scenario and is blocked on a testbed fixture, not on the suite.
  • Hooks and caching are not covered; @caching is reserved.

Answers #417's Q1 (directory layout, forced rather than chosen) and Q4 (numeric coercion stays a capability). Q2, Q3, Q5 and Q6 stay open; Q7 is split into #424.

@aepfli
aepfli requested a review from a team as a code owner October 1, 2026 07:54
@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

@aepfli
aepfli marked this pull request as draft October 2, 2026 08:41
@aepfli
aepfli force-pushed the feat/provider-tck-appendix branch from 8374621 to 9a1b36d Compare October 2, 2026 08:46
@aepfli
aepfli force-pushed the feat/provider-tck-appendix-doc branch from bbe6465 to 1d7b841 Compare October 2, 2026 08:46
@aepfli
aepfli force-pushed the feat/provider-tck-appendix branch from 9a1b36d to 068f936 Compare October 7, 2026 12:58
@aepfli
aepfli force-pushed the feat/provider-tck-appendix-doc branch from 1d7b841 to 78288e6 Compare October 7, 2026 12:58
toddbaert added a commit that referenced this pull request Oct 7, 2026
> Reopens [#431](#431), which
could not be reopened after its branch moved. Same change, plus a second
package. **Base of the provider-conformance stack**:
[#423](#423) (assets) →
[#439](#439) (Appendix F) →
[#425](#425) (report schema)
stack on this, but it stands on its own and can merge alone.

## What

Sets up [release-please](https://github.com/googleapis/release-please)
to cut the specification's releases, replacing the manual tagging step.

## Why

The `vX.Y.Z` tags are created by hand. That is easy to postpone — v0.9.0
went out in July and the commits since it are unreleased — and it
assembles the changelog from whoever's memory is handy. release-please
proposes the version and the notes as a pull request, and tags once that
pull request is merged.

## What does not change

| | before | after |
|---|---|---|
| tag | `v0.9.0` | `v0.9.1`, `v0.10.0`, … |
| next version after a breaking change | minor, while < 1.0.0 |
unchanged |
| who decides a release goes out | a maintainer merging | a maintainer
merging the release PR |

`include-component-in-tag: false` keeps the bare `vX.Y.Z` shape rather
than `spec/vX.Y.Z`, and `.release-please-manifest.json` starts at
`0.9.0` so numbering continues rather than restarting. `release-type:
simple` because the root `package.json` is `private: true` and carries
no version of its own; `simple` tracks the version in `version.txt`,
which is why that file is added here.

## Scope: the specification package only

One package, `.`, tagged `vX.Y.Z` as it always has been.

The conformance assets under `specification/assets/provider-tck` need
their own tags too — a Go consumer cannot name a version without one —
but that configuration belongs beside the assets, so it is added in
[#423](#423) along with the
files it describes. This PR stays the general setup and can merge on its
own.

## What does change, and needs a decision

**Release notes become generated from conventional commits.** v0.9.0's
notes were written by hand, including the `⚠️ This release contains
breaking changes` callout linking #280, #306 and #360. Anything beyond
the commit subjects now has to be added to the release pull request
before it is merged.

If the TSC would rather keep the old shape, `changelog-sections` can be
tuned in a follow-up — the defaults are left here so the first release
is easy to eyeball.

## What to expect on merge

release-please runs on push to `main` and should open a release pull
request proposing **v0.9.1**; the non-chore commits since v0.9.0 are
#416, #419, #429 and #408, all `fix:` or `docs:`. Nothing is tagged
until that pull request is merged, so the first run is reviewable before
it has any effect.

The action configuration is the shape v5 actually reads: `command` and
`default-branch` are not v5 inputs, `target-branch` is, and `signoff` is
a configuration-file option rather than an action input — which matters
because this repository enforces DCO on the bot's own commits.

---------

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Signed-off-by: Todd Baert <todd.baert@dynatrace.com>
Co-authored-by: Todd Baert <todd.baert@dynatrace.com>
@aepfli
aepfli force-pushed the feat/provider-tck-appendix branch from 068f936 to 347974f Compare October 8, 2026 10:29
@aepfli
aepfli force-pushed the feat/provider-tck-appendix-doc branch from 78288e6 to 58c475d Compare October 8, 2026 10:29
@toddbaert
toddbaert force-pushed the feat/provider-tck-appendix branch 2 times, most recently from 2f17331 to 986958f Compare October 13, 2026 18:33
@aepfli
aepfli force-pushed the feat/provider-tck-appendix-doc branch from 58c475d to ec4bb2f Compare October 14, 2026 13:07
Base automatically changed from feat/provider-tck-appendix to main October 14, 2026 14:05
toddbaert pushed a commit that referenced this pull request Oct 14, 2026
> **Part of [#417](#417
This PR is the **assets** — the files four conformance implementations
execute. It stacks on
[#440](#440), which configures
the releases that give these assets a tag. The normative description of
them, Appendix F, is
[#439](#439), stacked on this.
[#425](#425) adds the
machine-readable report.
>
> This PR previously carried the appendix too. It was split so the
assets can land while the prose is still being reviewed.

## What

The artifacts a provider conformance suite runs against:

| path | what it is |
|---|---|
| `assets/provider-tck/gherkin/` | six feature files — evaluation,
errors, events, lifecycle, reason, metadata |
| `assets/provider-tck/flags/canonical-flags.json` | the flag set every
scenario assumes |
| `assets/provider-tck/openapi/control-api.yaml` | the HTTP surface a
backend under test must expose |
| `assets/provider-tck/go.mod`, `embed.go` | a nested Go module exposing
the artifacts as an `embed.FS` |
| `release-please-config.json`, `.release-please-manifest.json` |
registers the assets as a releasable component; the general setup is
#440 |

Marked **experimental**: the scenario set is a representative subset,
and the capability vocabulary may still change.

## Why this is split out

Four language implementations already run these files, and all four are
currently pinned to an **unmerged branch tip**. Every rebase here
invalidates them — it has happened three times in two weeks. Nothing
about those pins becomes real until the assets are on `main` with a tag,
so landing the executable half first is worth more than landing it
together with prose that will take longer to review.

Appendix F is the same work, reviewed on its own terms.

## Why a Go module is in here

A Go module is distributed as a zip of the VCS tree, in which a git
submodule is only a gitlink and its files are absent. The other three
languages build from a working tree and keep using the submodule, but a
Go suite would have to vendor a copy of every artifact and police it
against drift. `embed.go` removes the copy: a consumer pins a revision,
the checksum database makes it immutable, and the embedded bytes are the
same bytes every other language reads out of the submodule. `go.mod` and
`embed.go` are inert for every non-Go consumer. Precedent:
`open-feature/flagd-schemas`.

The release config tags the assets as their own component,
`specification/assets/provider-tck/vX.Y.Z` — the shape `go get` needs.
Without it a Go consumer can only name a pseudo-version.

## Releasing the assets

The general release-please setup is
[#440](#440), which this stacks
on. What this PR adds is the part that describes *these* assets: a
second package, so they are tagged
`specification/assets/provider-tck/vX.Y.Z`.

That tag shape is what `go get` needs. Without it a Go consumer can only
name a pseudo-version, which is what the four conformance
implementations have had to do while this branch was unmerged, and it
has cost several rounds of re-pinning.

The specification package sets `exclude-paths` for this directory, so an
asset-only change no longer cuts a specification release, and
`separate-pull-requests` keeps the two from sharing one release pull
request.

## The capability vocabulary lives in the README, for now

A scenario exercising an optional part of the contract carries a tag; a
provider declares what it supports, and a scenario gated on an
undeclared tag is reported **skipped, with the reason** — never as
passed. A suite that quietly goes green on scenarios it did not run is
worse than no suite at all.

That rule and the tag table are in `assets/provider-tck/README.md` so
the tags in the feature files are not left undefined while the appendix
is in review. #439 moves them into the appendix and leaves a pointer
behind, so they end up with one home rather than two that can disagree.

Worth stating plainly: **withholding a tag is not an admission of a
defect.** `@numeric-coercion` borrows its rule from flagd's coercion
ADR, and `@string-typing` / `@fully-typed-values` sit on a question the
specification does not answer at all —
[#430](#430) and
[#433](#433).

## Decisions worth reviewing

- **`assets/provider-tck/` rather than beside the SDK Gherkin**, because
`assets/gherkin/evaluation.feature` already exists and tests an SDK, not
a provider. This answers #417's Q1 — the subdirectory is forced rather
than chosen.
- **`.gitattributes` normalises these to LF.** Four languages consume
them byte for byte and copy them verbatim into published artifacts, so a
Windows checkout must not package differently from a Linux one.
- **`missing-flag` must not exist** in the flag set — its absence is
what the `FLAG_NOT_FOUND` scenario tests. The other four load-bearing
properties of the flag set are in the README; each one, if seeded wrong,
turns a scenario green or red for the wrong reason in every language at
once.
- **No container restarts.** Outages are produced through the control
API, because orchestrators reassign host ports on restart and every
provider pointed at the old one then fails looking flaky.

## Implementations

All four execute exactly these files, against flagd (both resolvers),
OFREP and Flagsmith:

| | suite | flagd | OFREP | Flagsmith |
|---|---|---|---|---|
| Java |
[#1830](open-feature/java-sdk-contrib#1830) |
[#1847](open-feature/java-sdk-contrib#1847) |
[#1840](open-feature/java-sdk-contrib#1840) |
[#1849](open-feature/java-sdk-contrib#1849) |
| Go | [#940](open-feature/go-sdk-contrib#940) |
[#941](open-feature/go-sdk-contrib#941) |
[#942](open-feature/go-sdk-contrib#942) |
[#959](open-feature/go-sdk-contrib#959) |
| JavaScript |
[#1606](open-feature/js-sdk-contrib#1606) |
[#1608](open-feature/js-sdk-contrib#1608) |
[#1612](open-feature/js-sdk-contrib#1612) |
[#1623](open-feature/js-sdk-contrib#1623) |
| Python |
[#409](open-feature/python-sdk-contrib#409) |
[#411](open-feature/python-sdk-contrib#411) |
[#414](open-feature/python-sdk-contrib#414) | —
|

## What running them has found

- **flagd silently narrows a float flag to an integer**, Go and Java,
both resolvers, no error code —
[flagd#1996](open-feature/flagd#1996)
- **flagd reported ready before flags were evaluable**, on 100% of
starts —
[flagd#2047](open-feature/flagd#2047), fixed
in flagd v0.17.0
- **The Java SDK's `MultiProvider` swallows child provider events** —
[java-sdk#1882](open-feature/java-sdk#1882)
- **The Python flagd provider widens a boolean to a float** —
[python-sdk-contrib#417](open-feature/python-sdk-contrib#417)
- **The JavaScript Flagsmith provider stringifies unconditionally**
where its other paths correctly report a mismatch — visible only because
three languages ran identical scenarios against one backend
- two specification gaps, raised rather than decided here:
[#430](#430),
[#433](#433)

---------

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The normative description of the assets added alongside: what a TCK
implementation must do, what each capability asserts and when withholding one is
a sanctioned choice rather than a defect, the control-API contract, the
extension point, and the run-integrity checks.

Stated as normative blockquotes rather than numbered requirements, following
Appendix A -- specification.json is built from the numbered sections and no
appendix contributes to it. Whether any of this should become numbered is a TSC
decision, recorded as an open question.

The assets README hands the capability vocabulary back here, so it has one home
rather than two that can disagree.

Experimental and non-normative. Tracked in #417.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
@aepfli
aepfli force-pushed the feat/provider-tck-appendix-doc branch from ec4bb2f to a207701 Compare October 14, 2026 14:20
toddbaert pushed a commit that referenced this pull request Oct 16, 2026
Stops [#447](#447) from cutting
the conformance assets as **1.0.0**.

## Why it proposed 1.0.0

`.release-please-manifest.json` carries
`"specification/assets/provider-tck": "0.0.0"`. release-please reads
`0.0.0` as *no release yet*, so it never bumps from it — it falls back
to its default initial version, which is `1.0.0`. `bump-minor-pre-major:
true` is already set and had no effect here, because there is no prior
version to bump minor-instead-of-major.

## Why 1.0.0 is the wrong number

It is a compatibility claim, and `go get` is where it gets read.

- **Appendix F, which gives these assets their meaning, is not merged**
([#439](#439)) and marks itself
experimental.
- The **capability vocabulary** is still under review, and tags are the
suite's public surface: adding, splitting or renaming one changes what
every adopter's run means. `@string-typing` was split out of
`@fully-typed-values` during review, which is exactly the kind of change
1.0.0 promises not to make.
- The **control-API surface** (`openapi/control-api.yaml`) and the
**report schema**
([#425](#425)) are both open
questions.
- [#430](#430) and
[#433](#433) are open gaps in
what the assets assert.

## What this changes

One line: `initial-version: "0.1.0"` on the assets package.

`0.1.0` rather than `0.0.1` because the first release is a feature, not
a fix, and because it leaves both digits meaningful — patch for fixture
corrections, minor for new scenarios. Both are expected: the testbed
already owes two fixtures
([flagd-testbed#392](open-feature/flagd-testbed#392)),
and the open questions above will add scenarios.

## What to expect on merge

release-please re-runs on push to `main` and should update #447 to
propose **0.1.0**, tagging `specification/assets/provider-tck/v0.1.0`
once that is merged. Nothing is tagged by this PR.

Four conformance implementations currently pin these assets by commit,
because no tag exists yet. That tag is what lets Go name a version
instead of a pseudo-version, so the number it carries is worth getting
right before the first one is published — a tag is not practical to take
back.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant