The problem

OpenFeature's central promise is vendor neutrality: swapping providers must not change application
behaviour. Nothing currently verifies that.

Every provider tests differently. Some have thorough suites, some have almost none, and no two agree
on what "implements the provider contract" means. The consequences are real and already visible:

• Providers disagree on whether a type mismatch returns the code default or throws.
• Providers disagree on whether an unreachable backend yields ERROR promptly, hangs, or throws out
  of provider registration.
• Providers disagree on which lifecycle events fire, and in what order, around a disconnect.
• Some providers silently coerce between numeric types instead of reporting TYPE_MISMATCH.

Every one of these is invisible until an application swaps providers in production. The spec
describes the correct behaviour in prose; there is no executable definition of it.

This is the tracking issue. The design lives in Appendix F
(#423) — the three artifacts, the control API,
the capability vocabulary, the extension rules and the remaining open questions are all specified
there, and that is the copy to read and to argue with. This issue carries status, per-language
links and what the suite has found.

What this is

A language-agnostic conformance suite for providers: Gherkin scenarios, a standardised backend
control API and a canonical flag set, living in this repository so that every language's TCK runs
the same definition of conformance rather than inventing its own. Each language packages those
artifacts and supplies the runner — step definitions, the container lifecycle, a control-API client
— in whatever form its testing culture expects. It verifies the provider contract only: typed
resolution, keeping integer and float distinct, error handling, lifecycle and events. Backend
evaluation logic, targeting correctness and the provider↔backend wire protocol are all out of scope.

Status

All four languages have the same shape — a suite PR off main, forking into a flagd adoption with
an OFREP adoption stacked on it, and a conformance report PR — on matching branch names
(feat/provider-tck, then -flagd, -ofrep, -report). Nothing has merged yet; everything is in
draft pending review of #423.

All four pin the spec at the same commit and read the three artifacts from that pin rather than from
a copy, so no language can drift from the definition of conformance without the pin moving.

Decisions settled across all four languages

Each of these began as four different answers, and several were only visible because there were
four implementations to compare:

• The package is tck in every language — dev.openfeature.contrib.tools:tck,
  github.com/open-feature/go-sdk-contrib/tools/tck, @openfeature/tck, openfeature-tck — all at
  0.1.0. It was provider-tck everywhere; renamed at
  @erka's request,
  since the artifact is not a provider and a flat tck leaves room for a suite that tests something
  other than one.
• The suite owns the container stack. An adopter supplies a Compose file, the ports the provider
  connects to, and a factory taking a discovered endpoint; everything else — starting the stack,
  discovering dynamically mapped ports, building the control client, waiting for the control API,
  teardown — belongs to the TCK. Three of four languages originally shipped only the control-API
  client, and each adoption then hand-rolled between 130 and 460 lines of the same wrapper. Now
  required by Appendix F.
• knownDeviations has one meaning. Two legitimate shapes, with declare-and-let-it-fail
  preferred over withhold-and-skip, summary required and issue optional. Previously Java and Go
  documented one shape, JS the other and called the first discouraged, and Python required an issue
  link with no untracked form — one field read three ways.
• controlApi is required and closed-typed, stated by the control rather than inferred by the
  harness. It was a defaulting method in Java, an optional field in JS, a duck-typed property in
  Python and absent from Go's interface. Nothing outside the control can tell which path a run used,
  and an absent value is not neutral: every run is one or the other.
• Every state-changing control endpoint must serve the new state before returning. /start
  already said so; /change and /reset did not, and /reset is the endpoint a suite calls before
  every scenario — so a window there is re-rolled per scenario rather than once per suite.
• /restart is optional. It was [REQUIRED] on the strength of a claim in its own description
  that the TCK used it for disconnect/reconnect. It does not: that scenario is an unbounded outage,
  served by /stop + /start, and nothing in the shipped Gherkin reaches /restart in any language.
• The resolution reason is a claim, not a house rule. The suite asserted an exact reason in
  thirteen places across three feature files, narrowing a SHOULD into a MUST for every adopter.
  Those are gone; the reasons now live in one gated reason.feature, and declaring
  @standard-reasons means "I use the standard vocabulary with the standard meanings". A provider
  whose backend reports vendor-specific reasons simply does not declare it and loses nothing — its
  values, variants and error codes are asserted on MUSTs regardless. This also lets Appendix F
  state what each reason means without asking the specification to close the set, since the mapping
  is the content of an opt-in claim.
• A capability the language's SDK cannot express is refused by the implementation, not left to
  adopters. Two exist: @large-integers where the integer accessor is 32-bit, @numeric-coercion
  where the language has one numeric type. Neither says anything about a provider. Before this,
  JavaScript restated that fact in five places and Java in five — every one a chance to put a claim
  in a report that no scenario could verify.
• An adoption suite is not a required CI gate while real gaps remain, because its honest output
  is red and making it block a merge forces someone to silence it. All four languages believed their
  adoption suites were excluded from CI; all four were running them, red and unwatched, each
  defeated by a different mechanism — a profile that cleared the exclusion, a build tag applied to
  every module, a default test task that simply collected them, and a run-all target.

Per-language tracking

Not started: .NET, PHP, Ruby, Kotlin/Swift (client-side). JavaScript is server-SDK only so far, so a
web-SDK provider cannot adopt it yet.

OFREP adoption is cheap in any language that has an OFREP provider: flagd already serves OFREP on
port 8016 and flagd-testbed's compose exposes it, so the existing testbed and launchpad control
API drive it with no new infrastructure.

Backends under test

Three testbeds implement the control API. The rule for each: fake the config source, never the evaluation.

flagd-testbed needed four changes, which came out of this work:

#394 and #395 together gate the most, and they are one defect seen from two ends. The launchpad
404s on POST /reset, so a suite that prefers /reset for scenario isolation falls back to
POST /start?config=default before every scenario — a full flagd restart each time. And /start
returns before the file source has loaded. So a race measured at ~40 ms and 35% of starts is
re-rolled per scenario instead of once per suite, which is why two runs of the same suite disagree
about which scenarios failed. Measured on one adoption at 5 to 41 failures out of 47 across five
runs, and equally bad against the hand-rolled container wrapper it replaced, so it is the backend
and not the harness. #394 makes /start honest; #395 removes the need to restart at all.

Findings from the language implementations

Bugs the suite surfaced, listed because the point of the exercise is whether it finds real things.

Four of these are worth dwelling on.

One defect was surfaced by a scenario that passed. The Python flagd provider's shutdown
completes, and within the bound the scenario asserts — and leaves a gRPC traceback behind it, because
the channel closes while the connectivity watcher is still polling it. Nothing required is unmet, so
the results record a pass and the report has nowhere to put it. It was found only because the
@lifecycle capability had never been declared in that adoption, so those six scenarios had never
run at all; declaring it was a one-line change. Two lessons: an undeclared capability is a coverage
gap that compounds silently, and a conformance report is not the whole of what running the suite
tells you.

The flagd resolvers report an outage differently from each other, which is precisely the
vendor-neutrality claim the suite is meant to check.

The Python bool bugs could not have been found by any other language's suite — bool is a
subclass of int only in Python. That is the clearest argument yet for implementing this in more
than one language rather than treating one implementation as sufficient. Having been caught twice,
once per numeric accessor, it is also an argument for the suite asserting both directions.

Three findings are merged and carried by no release. python-sdk#619, python-sdk#620 and
go-sdk#552 are all fixed on main while the newest releases predate the fixes, so both suites still
resolve pre-fix SDKs, still ship a wrapper class, and still hold a strict xfail. Each needs a pin
bump and a marker removed rather than further investigation — worth tracking as a release question
rather than as three open bugs.

flagd#2047 needs no provider, SDK or TCK to
reproduce — plain flagd, one file source, a tight loop on /readyz. It surfaced because the suite
has to wait for readiness before it can evaluate anything, which turned "when is a backend ready?"
into a question someone had to answer precisely. flagd is not violating its documented contract; the
claim is that "one successful data sync" is the wrong semantics for a readiness probe, because
delivered is not applied.

Open questions

Appendix F carries the full set; these are the ones that most want other people's opinions.

1. Should reason be an open set at all? The suite no longer depends on the answer — it asks
   only providers that declare @standard-reasons — but the question stands on its own.
   Requirement 2.2.5
   permits "some other string", while the OpenTelemetry convention that consumes reason is closed
   at exactly the nine OpenFeature values, and Appendix D already instructs integrations to treat it
   as an enumeration and transform it. So a vendor-specific reason has nowhere to land downstream.
   Worth knowing before deciding: how many providers actually emit one today. Every conformance
   report now answers that for its provider, which turns the question into something measurable
   rather than a matter of opinion.
2. A report's declaration no longer explains every skip, and consumers should know it. A
   capability may be absent because the provider declined it or because the language's SDK cannot
   express it, and the two are indistinguishable from the declaration alone. The results payload
   carries the distinction per skip and Appendix F records which capabilities are affected per
   language, but anything comparing providers across languages has to read the skip reason rather
   than infer from the declaration.
3. Nothing validates an emitted report against the schema in CI, in any language — each checks
   fields by hand or validates out of band. The schema is what makes reports comparable across
   languages, so this is the weakest link in the design, and it wants one answer rather than four.
4. What a provider must do when a number does not fit the accessor —
   #430, which @numeric-coercion currently gates
   rather than answers.

Prior art in this ecosystem

• open-feature/test-harness — the flagd-oriented
  Gherkin suite and testbed this work generalises.
• open-feature/flagd-testbed — the launchpad
  control API the standardised one is derived from.
• dev.openfeature.contrib.tools:flagd-api-testkit in java-sdk-contrib — an existing "abstract
  suite + SPI factory, features packaged in the JAR" testkit whose adoption ergonomics the Java
  implementation copies.