{{ message }}
Repository navigation
Conversation
This was referenced Oct 1, 2026
aepfli
marked this pull request as draft
October 2, 2026 08:41
aepfli
force-pushed
the
feat/provider-tck-appendix
branch
from
October 2, 2026 08:46
8374621 to
9a1b36d
Compare
aepfli
force-pushed
the
feat/provider-tck-appendix-doc
branch
from
October 2, 2026 08:46
bbe6465 to
1d7b841
Compare
aepfli
force-pushed
the
feat/provider-tck-appendix
branch
from
October 7, 2026 12:58
9a1b36d to
068f936
Compare
aepfli
force-pushed
the
feat/provider-tck-appendix-doc
branch
from
October 7, 2026 12:58
1d7b841 to
78288e6
Compare
toddbaert
added a commit
that referenced
this pull request
Oct 7, 2026
> Reopens [#431](#431), which could not be reopened after its branch moved. Same change, plus a second package. **Base of the provider-conformance stack**: [#423](#423) (assets) → [#439](#439) (Appendix F) → [#425](#425) (report schema) stack on this, but it stands on its own and can merge alone. ## What Sets up [release-please](https://github.com/googleapis/release-please) to cut the specification's releases, replacing the manual tagging step. ## Why The `vX.Y.Z` tags are created by hand. That is easy to postpone — v0.9.0 went out in July and the commits since it are unreleased — and it assembles the changelog from whoever's memory is handy. release-please proposes the version and the notes as a pull request, and tags once that pull request is merged. ## What does not change | | before | after | |---|---|---| | tag | `v0.9.0` | `v0.9.1`, `v0.10.0`, … | | next version after a breaking change | minor, while < 1.0.0 | unchanged | | who decides a release goes out | a maintainer merging | a maintainer merging the release PR | `include-component-in-tag: false` keeps the bare `vX.Y.Z` shape rather than `spec/vX.Y.Z`, and `.release-please-manifest.json` starts at `0.9.0` so numbering continues rather than restarting. `release-type: simple` because the root `package.json` is `private: true` and carries no version of its own; `simple` tracks the version in `version.txt`, which is why that file is added here. ## Scope: the specification package only One package, `.`, tagged `vX.Y.Z` as it always has been. The conformance assets under `specification/assets/provider-tck` need their own tags too — a Go consumer cannot name a version without one — but that configuration belongs beside the assets, so it is added in [#423](#423) along with the files it describes. This PR stays the general setup and can merge on its own. ## What does change, and needs a decision **Release notes become generated from conventional commits.** v0.9.0's notes were written by hand, including the `⚠️ This release contains breaking changes` callout linking #280, #306 and #360. Anything beyond the commit subjects now has to be added to the release pull request before it is merged. If the TSC would rather keep the old shape, `changelog-sections` can be tuned in a follow-up — the defaults are left here so the first release is easy to eyeball. ## What to expect on merge release-please runs on push to `main` and should open a release pull request proposing **v0.9.1**; the non-chore commits since v0.9.0 are #416, #419, #429 and #408, all `fix:` or `docs:`. Nothing is tagged until that pull request is merged, so the first run is reviewable before it has any effect. The action configuration is the shape v5 actually reads: `command` and `default-branch` are not v5 inputs, `target-branch` is, and `signoff` is a configuration-file option rather than an action input — which matters because this repository enforces DCO on the bot's own commits. --------- Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com> Signed-off-by: Todd Baert <todd.baert@dynatrace.com> Co-authored-by: Todd Baert <todd.baert@dynatrace.com>
aepfli
force-pushed
the
feat/provider-tck-appendix
branch
from
October 8, 2026 10:29
068f936 to
347974f
Compare
aepfli
force-pushed
the
feat/provider-tck-appendix-doc
branch
from
October 8, 2026 10:29
78288e6 to
58c475d
Compare
toddbaert
force-pushed
the
feat/provider-tck-appendix
branch
2 times, most recently
from
October 13, 2026 18:33
2f17331 to
986958f
Compare
aepfli
force-pushed
the
feat/provider-tck-appendix-doc
branch
from
October 14, 2026 13:07
58c475d to
ec4bb2f
Compare
toddbaert
pushed a commit
that referenced
this pull request
Oct 14, 2026
> **Part of [#417](#417 This PR is the **assets** — the files four conformance implementations execute. It stacks on [#440](#440), which configures the releases that give these assets a tag. The normative description of them, Appendix F, is [#439](#439), stacked on this. [#425](#425) adds the machine-readable report. > > This PR previously carried the appendix too. It was split so the assets can land while the prose is still being reviewed. ## What The artifacts a provider conformance suite runs against: | path | what it is | |---|---| | `assets/provider-tck/gherkin/` | six feature files — evaluation, errors, events, lifecycle, reason, metadata | | `assets/provider-tck/flags/canonical-flags.json` | the flag set every scenario assumes | | `assets/provider-tck/openapi/control-api.yaml` | the HTTP surface a backend under test must expose | | `assets/provider-tck/go.mod`, `embed.go` | a nested Go module exposing the artifacts as an `embed.FS` | | `release-please-config.json`, `.release-please-manifest.json` | registers the assets as a releasable component; the general setup is #440 | Marked **experimental**: the scenario set is a representative subset, and the capability vocabulary may still change. ## Why this is split out Four language implementations already run these files, and all four are currently pinned to an **unmerged branch tip**. Every rebase here invalidates them — it has happened three times in two weeks. Nothing about those pins becomes real until the assets are on `main` with a tag, so landing the executable half first is worth more than landing it together with prose that will take longer to review. Appendix F is the same work, reviewed on its own terms. ## Why a Go module is in here A Go module is distributed as a zip of the VCS tree, in which a git submodule is only a gitlink and its files are absent. The other three languages build from a working tree and keep using the submodule, but a Go suite would have to vendor a copy of every artifact and police it against drift. `embed.go` removes the copy: a consumer pins a revision, the checksum database makes it immutable, and the embedded bytes are the same bytes every other language reads out of the submodule. `go.mod` and `embed.go` are inert for every non-Go consumer. Precedent: `open-feature/flagd-schemas`. The release config tags the assets as their own component, `specification/assets/provider-tck/vX.Y.Z` — the shape `go get` needs. Without it a Go consumer can only name a pseudo-version. ## Releasing the assets The general release-please setup is [#440](#440), which this stacks on. What this PR adds is the part that describes *these* assets: a second package, so they are tagged `specification/assets/provider-tck/vX.Y.Z`. That tag shape is what `go get` needs. Without it a Go consumer can only name a pseudo-version, which is what the four conformance implementations have had to do while this branch was unmerged, and it has cost several rounds of re-pinning. The specification package sets `exclude-paths` for this directory, so an asset-only change no longer cuts a specification release, and `separate-pull-requests` keeps the two from sharing one release pull request. ## The capability vocabulary lives in the README, for now A scenario exercising an optional part of the contract carries a tag; a provider declares what it supports, and a scenario gated on an undeclared tag is reported **skipped, with the reason** — never as passed. A suite that quietly goes green on scenarios it did not run is worse than no suite at all. That rule and the tag table are in `assets/provider-tck/README.md` so the tags in the feature files are not left undefined while the appendix is in review. #439 moves them into the appendix and leaves a pointer behind, so they end up with one home rather than two that can disagree. Worth stating plainly: **withholding a tag is not an admission of a defect.** `@numeric-coercion` borrows its rule from flagd's coercion ADR, and `@string-typing` / `@fully-typed-values` sit on a question the specification does not answer at all — [#430](#430) and [#433](#433). ## Decisions worth reviewing - **`assets/provider-tck/` rather than beside the SDK Gherkin**, because `assets/gherkin/evaluation.feature` already exists and tests an SDK, not a provider. This answers #417's Q1 — the subdirectory is forced rather than chosen. - **`.gitattributes` normalises these to LF.** Four languages consume them byte for byte and copy them verbatim into published artifacts, so a Windows checkout must not package differently from a Linux one. - **`missing-flag` must not exist** in the flag set — its absence is what the `FLAG_NOT_FOUND` scenario tests. The other four load-bearing properties of the flag set are in the README; each one, if seeded wrong, turns a scenario green or red for the wrong reason in every language at once. - **No container restarts.** Outages are produced through the control API, because orchestrators reassign host ports on restart and every provider pointed at the old one then fails looking flaky. ## Implementations All four execute exactly these files, against flagd (both resolvers), OFREP and Flagsmith: | | suite | flagd | OFREP | Flagsmith | |---|---|---|---|---| | Java | [#1830](open-feature/java-sdk-contrib#1830) | [#1847](open-feature/java-sdk-contrib#1847) | [#1840](open-feature/java-sdk-contrib#1840) | [#1849](open-feature/java-sdk-contrib#1849) | | Go | [#940](open-feature/go-sdk-contrib#940) | [#941](open-feature/go-sdk-contrib#941) | [#942](open-feature/go-sdk-contrib#942) | [#959](open-feature/go-sdk-contrib#959) | | JavaScript | [#1606](open-feature/js-sdk-contrib#1606) | [#1608](open-feature/js-sdk-contrib#1608) | [#1612](open-feature/js-sdk-contrib#1612) | [#1623](open-feature/js-sdk-contrib#1623) | | Python | [#409](open-feature/python-sdk-contrib#409) | [#411](open-feature/python-sdk-contrib#411) | [#414](open-feature/python-sdk-contrib#414) | — | ## What running them has found - **flagd silently narrows a float flag to an integer**, Go and Java, both resolvers, no error code — [flagd#1996](open-feature/flagd#1996) - **flagd reported ready before flags were evaluable**, on 100% of starts — [flagd#2047](open-feature/flagd#2047), fixed in flagd v0.17.0 - **The Java SDK's `MultiProvider` swallows child provider events** — [java-sdk#1882](open-feature/java-sdk#1882) - **The Python flagd provider widens a boolean to a float** — [python-sdk-contrib#417](open-feature/python-sdk-contrib#417) - **The JavaScript Flagsmith provider stringifies unconditionally** where its other paths correctly report a mismatch — visible only because three languages ran identical scenarios against one backend - two specification gaps, raised rather than decided here: [#430](#430), [#433](#433) --------- Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The normative description of the assets added alongside: what a TCK implementation must do, what each capability asserts and when withholding one is a sanctioned choice rather than a defect, the control-API contract, the extension point, and the run-integrity checks. Stated as normative blockquotes rather than numbered requirements, following Appendix A -- specification.json is built from the numbered sections and no appendix contributes to it. Whether any of this should become numbered is a TSC decision, recorded as an open question. The assets README hands the capability vocabulary back here, so it has one home rather than two that can disagree. Experimental and non-normative. Tracked in #417. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
force-pushed
the
feat/provider-tck-appendix-doc
branch
from
October 14, 2026 14:20
ec4bb2f to
a207701
Compare
This was referenced Oct 14, 2026
toddbaert
pushed a commit
that referenced
this pull request
Oct 16, 2026
Stops [#447](#447) from cutting the conformance assets as **1.0.0**. ## Why it proposed 1.0.0 `.release-please-manifest.json` carries `"specification/assets/provider-tck": "0.0.0"`. release-please reads `0.0.0` as *no release yet*, so it never bumps from it — it falls back to its default initial version, which is `1.0.0`. `bump-minor-pre-major: true` is already set and had no effect here, because there is no prior version to bump minor-instead-of-major. ## Why 1.0.0 is the wrong number It is a compatibility claim, and `go get` is where it gets read. - **Appendix F, which gives these assets their meaning, is not merged** ([#439](#439)) and marks itself experimental. - The **capability vocabulary** is still under review, and tags are the suite's public surface: adding, splitting or renaming one changes what every adopter's run means. `@string-typing` was split out of `@fully-typed-values` during review, which is exactly the kind of change 1.0.0 promises not to make. - The **control-API surface** (`openapi/control-api.yaml`) and the **report schema** ([#425](#425)) are both open questions. - [#430](#430) and [#433](#433) are open gaps in what the assets assert. ## What this changes One line: `initial-version: "0.1.0"` on the assets package. `0.1.0` rather than `0.0.1` because the first release is a feature, not a fix, and because it leaves both digits meaningful — patch for fixture corrections, minor for new scenarios. Both are expected: the testbed already owes two fixtures ([flagd-testbed#392](open-feature/flagd-testbed#392)), and the open questions above will add scenarios. ## What to expect on merge release-please re-runs on push to `main` and should update #447 to propose **0.1.0**, tagging `specification/assets/provider-tck/v0.1.0` once that is merged. Nothing is tagged by this PR. Four conformance implementations currently pin these assets by commit, because no tag exists yet. That tag is what lets Go name a version instead of a pseudo-version, so the number it carries is worth getting right before the first one is published — a tag is not practical to take back. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What
Appendix F: Provider Conformance (TCK) — the normative description of the conformance assets now on
main(#423). What a TCK implementation must do, what each capability asserts, the control-API contract, the extension point, and the run-integrity checks.Marked experimental and non-normative.
How the requirements are expressed
As normative blockquotes (58 of them), not numbered requirements — following Appendix A, since
specification.jsonis built from the numbered sections and no appendix contributes to it.Whether any of this should become numbered normative sections is a TSC decision, recorded as an open question rather than assumed. That is the main thing I would like a view on.
Design decisions worth reviewing
Each is argued in the appendix; this is the index.
@lifecycleand@eventsare separate, because a provider can initialise against its backend without emitting events, and vice versa.@numeric-coercionis borrowed, not specified. The rule comes from a flagd ADR, not from the specification. An earlier draft claimed the opposite and called withholding the tag "an admission of a known bug" — wrong, and corrected. Gap: #430.@string-typingand@fully-typed-valuesare two tags because one hid a bug. Three providers over one Flagsmith backend: two reportTYPE_MISMATCHfor a boolean and an integer asked as strings, one stringifies them, and all three stringify the float and the structure. The latter is the backend's shape, the former a provider defect; one tag reported the defect as a permitted absence. Gap: #433.One home for the vocabulary
assets/provider-tck/README.mdcarries the capability table onmain, so the tags are defined while this is in review. This PR moves it here and leaves a pointer behind — the appendix owns it, and two copies would be able to disagree.Open questions
@large-integerscovers up to 2^53 − 1; 32- versus 64-bit accessors are not modelled. Part of #430.@cachingis reserved.Answers #417's Q1 (directory layout, forced rather than chosen) and Q4 (numeric coercion stays a capability). Q2, Q3, Q5 and Q6 stay open; Q7 is split into #424.