Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host · Issue #424 · open-feature/spec · GitHub
Skip to content

Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host #424

Open
Open
Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host#424
Labels
enhancementNew feature or requestquestionFurther information is requestedroadmap-proposalThis initiative is suggested for the project roadmap

Description

@aepfli
aepfli
opened on Aug 24, 2026

Follow-up to #417, which asks in its open questions: "Who owns conformance claims? Is there a badge, a report format, a listing on openfeature.dev? Or is passing simply its own reward for now?" This issue is that question, split out so it can progress on its own timeline.

This is a design discussion, not a proposal to implement.

Status, 2026-09-11

The report format is no longer the substance of this issue. Following @toddbaert's comment below, #425 now defines only a small envelope and carries the results as Cucumber Messages rather than a format of our own. The sections below have been trimmed to what that leaves open: collection, trust and expiry. The minimized comments record how the format got there.

The opportunity

The provider conformance suite (Appendix F) has implementations in four languages, each of which already computes the data a public conformance report needs: the capability set a provider declares, and a per-scenario outcome. What is missing is a way to publish and collect reports. That would let openfeature.dev show, for any provider, what it actually supports — replacing "implements the provider contract" as an unverified claim with something a user can look up before choosing a provider.

The awkward part

Not every provider lives in an SDK-contrib repo. Many are in vendor organisations, on their own release cadence, and OpenFeature cannot run their suites — the backends and credentials are not ours. So any design has to work for providers we do not host and cannot execute.

Settled in #425

  • Comparability. A report records the TCK implementation, its version and the spec revision its assets came from. The revision is a property of the TCK artifact: every implementation consumes the assets from this repository, via git submodule or, for Go, via the nested Go module, so the pin is always known. For the feature files the revision is corroborated by the Messages stream, which carries the executed source.
  • Outcomes. Per-scenario outcomes come from Cucumber Messages. The envelope adds the declaration, which is an input to reading them, and a notApplicable set for capabilities a language cannot express — @numeric-coercion in JavaScript, which has no integer type. Collapsing that into "undeclared" would show every JS provider as missing something no JS provider can have.
  • Known deviations. A provider can say "failed, known bug, tracked here", which is distinct from "chose not to". Three implementations had each invented a local shape for this.
  • One provider, several reports. configuration keys them: flagd's RPC and in-process resolvers differ in whether they emit PROVIDER_STALE.

Never compute a conformance percentage

A single score rewards declaring fewer capabilities. Undeclare @stale and your percentage goes up. That is conformance theatre, and it exactly inverts the incentive the suite exists to create. A page should show a matrix of provider × capability with the distinct states visible, and no aggregate number. "This provider does not support configuration-change events" is genuinely useful to someone choosing a provider; hiding it behind a green 100% is actively harmful.

Open: trust

A JSON file in a vendor repository is a claim, not a fact. Three options:

Approach Verdict
OpenFeature re-runs every suite centrally does not scale — needs each vendor's backend and credentials
Naked self-report worthless the moment anything is contested
Self-report with provenance the only one that scales

Provenance meaning: the report is emitted only by CI, and carries the workflow run URL, commit SHA, and optionally a sigstore / GitHub OIDC attestation. The envelope has slots for all of these. Whatever we choose, the page should say plainly what it is showing: attested self-reports, not audits.

Open: discovery

Do not crawl. Suggested shape:

  • a registry of pointers in an OpenFeature-owned repo — provider → report URL, added once by a vendor PR
  • vendors self-publish the report to that URL (release asset, GitHub Pages, wherever)
  • a scheduled job fetches every registered URL, validates against the schema, and rebuilds the page

The human-reviewed list stays small and stable; report content refreshes without anyone raising a PR. A /.well-known/openfeature-conformance.json convention could layer on later for vendors who prefer it.

Open: expiry

@dgenio's comment below proposes separating subject supersession (a newer provider release exists) from contract supersession (a newer TCK asset set exists), and keeping old reports visible as historical rather than deleting them. That needs the envelope to identify the tested subject strongly enough to make the comparison; provider.version is currently optional.

Suggested phasing

  1. Schema in this repo. feat: add a schema for machine-readable provider conformance reports #425.
  2. Emitters in each language TCK. Go, Python and JavaScript emit the earlier draft format and will be updated to the envelope once its shape is agreed. Java is in progress.
  3. Registry + aggregator, designed once two or three real reports exist.
  4. The page, last.

Questions

  1. Should every TCK record the spec revision it vendored? Settled: the pin is always known. The envelope carries the commit and an optional release tag.
  2. Is not-applicable the right fourth state? Settled as declaration.notApplicable in feat: add a schema for machine-readable provider conformance reports #425. Whether the specification should additionally scope capabilities per language, so a language-wide impossibility is recorded once rather than in every report, is still worth deciding.
  3. Registry-of-pointers, or a well-known URL convention, or both?
  4. How much provenance is proportionate — a CI run URL, or a full signed attestation?
  5. Does a report expire? See the supersession model above.
  6. Who owns the registry and the aggregator — this repo, openfeature.dev, or something new?
  7. Should a report be published inside the provider's released artifact, and how strongly must the tested subject revision be identified?

Activity

  1. aepfli
    added a commit that references this issue on Aug 24, 2026

    feat: add a schema for machine-readable provider conformance reports

    ae42853

  2. aepfli commented on Aug 24, 2026

    @aepfli
    aepfli
    on Aug 24, 2026 · Hidden as outdated
    Author

  3. aepfli commented on Aug 24, 2026

    @aepfli
    aepfli
    on Aug 24, 2026 · Hidden as outdated
    Author

  4. aepfli commented on Aug 24, 2026

    @aepfli
    aepfli
    on Aug 24, 2026 · Hidden as resolved
    Author

  5. aepfli commented on Aug 24, 2026

    @aepfli
    aepfli
    on Aug 24, 2026
    MemberAuthor

    Decided: a Scenario Outline row is identified by its parameters

    scenarios[] entries were keyed on feature + name, and every row of an outline shares one name. The type-mismatch matrix is eleven rows, so a report carried eleven entries separated by nothing but a duration measurement — and the Python implementation produced exactly the case that makes this matter, with one of the eleven failing and ten passing under one key.

    An entry now carries the Examples row it came from, keyed by column header:

    {
  "feature": "errors",
  "name": "Requesting the wrong type returns the code default",
  "example": { "key": "boolean-flag", "requested": "Integer", "default": "1" },
  "outcome": "failed",
  "reason": "error-code was 'GENERAL', expected 'TYPE_MISMATCH'"
}

    Identity is (feature, name, example). Pushed to #425.

    Why a field rather than a naming convention

    The alternative was mandating a mangled name — name [key=boolean-flag, requested=Integer, default=1] — which needs no schema change and less work per implementation. It was rejected because it puts a separator, an ordering, and an escaping rule for values containing the separator into normative prose that every implementation has to reproduce byte-for-byte, and drift there stays invisible until two reports quietly fail to line up.

    That is not a hypothetical. The three implementations had already diverged on this point before the field existed, each locally reasonable and none compatible with the others:

    what landed in name for the same row
    Go the bare scenario name, for all eleven rows
    Python the name plus pytest's example id
    JavaScript jest-cucumber's expanded example title

    Values are the cell contents verbatim as strings, because Gherkin has no types — coercing "1" to a number would make the report disagree with the table it came from.

    Related, and worth noting for anyone writing an emitter

    JavaScript hit a second outline problem: jest-cucumber never applies a scenario-name template to outline rows, so four skipped rows were reported with no reason at all — a direct violation of Appendix F's "skipped, with the reason". Different bug, same root cause: outline rows are the case every runner treats as an afterthought. That is a good argument for the format being explicit about them rather than inheriting whatever each runner happens to do.

    All four implementations are being updated to populate the field.

  6. dgenio commented on Aug 24, 2026

    @dgenio
    dgenio
    on Aug 24, 2026

    I think the open question about whether a report "expires" becomes easier if we separate historical validity from current applicability.

    A conformance report is evidence about a particular tested subject against a particular contract:

    tested provider revision
× provider configuration
× SDK/runtime context where relevant
× TCK assets/spec revision
→ observed conformance result

    Time passing does not by itself make that evidence false.

    For example:

    provider v1.4 / rpc
tested against TCK assets A
→ passed

    If provider v1.5 is later released, the old report is still a valid statement about v1.4. It just is not evidence that v1.5 conforms.

    Likewise, if the TCK moves from assets A to assets B and adds or changes a mandatory scenario, the report against A remains historically valid but does not establish conformance to B.

    So I would avoid an expiresAt-style model and instead let the aggregator distinguish two independent forms of supersession:

    1. subject supersession — the provider implementation/release being discussed is no longer the one that was tested;
    2. contract supersession — the accepted/current TCK asset set is newer than the one exercised by the report.

    That also suggests one schema concern in #425: the report should always identify the tested subject strongly enough to make this comparison. provider.version is currently optional, while provenance.commit answers a different question: where the report came from. In some setups that commit may identify the provider source too, but that is not guaranteed.

    I would therefore keep these concepts separate:

    provider identity       = which provider/configuration
tested subject revision = which exact release/artifact/source revision was exercised
report provenance       = who/what produced the report and where
TCK identity            = which contract/questions were exercised

    I am not attached to requiring SemVer specifically. A provider version, source revision, artifact digest, or another stable provider-defined revision could all be reasonable depending on how the provider is distributed. The important invariant is that a report cannot silently float forward to a changed implementation.

    For presentation, I would keep old reports rather than deleting them:

    v1.4 / TCK A — passed (historical)
v1.5 / TCK B — no report / unverified

    rather than allowing the v1.4 result to appear to describe v1.5.

    This seems consistent with the rest of the design here: reports are claims with explicit provenance, not audits; configurations are not interchangeable; and absence/unsupported states should remain visible rather than being converted into reassuring aggregate numbers.

  7. aepfli
    added
    enhancementNew feature or request
    questionFurther information is requested
    roadmap-proposalThis initiative is suggested for the project roadmap
    on Aug 25, 2026

  8. toddbaert commented on Sep 9, 2026

    @toddbaert
    toddbaert
    on Sep 9, 2026
    Member

    I'd really love not to have to invent our own format for this. It's another thing to maintain and version, and there are standards out there already that cover most of what we need.

    Two worth considering before we commit to a bespoke schema:

    • Cucumber Messages (https://github.com/cucumber/messages) (the ndjson protocol (https://github.com/cucumber/messages#ndjson)) seems like it could work; it's maintained, cross-language, JSON-schema'd, and preserves per-scenario outcome, the tags (i.e. our capability), and a skip message. cucumber-jvm and godog emit it natively; we'd have to implement emitters for jest-cucumber and pytest-bdd, but that's implementing an existing spec rather than inventing one, and it would probably benefit others

    • JUnit/xUnit XML (common reference (https://github.com/testmoapp/junitxml)) is the universal format for this; every one of our four runners already emits it today. It's lossier (no tags, skip cause is free text), but we can carry the extra semantics with a conventions, I think.

    The key idea, either way: "not-declared" and "not-applicable" are just sub-categories of "skipped". So we don't need a format with four statuses; we just need a standard skip plus a way to communicate the reason, and I think both formats can carry that (Messages carries it with the tags, which is why I think that might be better, but more impl work).

    That leaves the only genuinely OpenFeature-specific piece as the capability summary, which is small and can be derived from the per-scenario results, instead of than being a concept we build... WDYT; is there a reason to prefer a custom format I'm missing?

  9. aepfli commented on Sep 10, 2026

    @aepfli
    aepfli
    on Sep 10, 2026
    MemberAuthor

    I fully agree and it makes sense. I will take a deeper look, this was for me just the next step, and i thought i add a feasibility to all my PRs for the TCK. I will explore the options, hopefully one of those are compatible with our whole stack or at least we can contribute upstream to make it work

  10. aepfli commented on Sep 10, 2026

    @aepfli
    aepfli
    on Sep 10, 2026
    MemberAuthor

    Sorry for the spam, i was not aware that my agent documented everything in the issues, I adapted my basic prompt and it should be better now.

  11. aepfli
    added 3 commits that reference this issue on Sep 10, 2026

    feat: carry results in Cucumber Messages instead of defining our own

    46ebc21

    feat: add a schema for machine-readable provider conformance reports

    f6593a3

    feat: carry results in Cucumber Messages instead of defining our own

    e8080b5

  12. 109 remaining items

  13. aepfli
    added 15 commits that reference this issue on Oct 8, 2026

    feat: carry results in Cucumber Messages instead of defining our own

    a537d85

    feat(tck): emit a machine-readable conformance report

    edebe4f

    feat(tck): emit a machine-readable conformance report

    4367637

    feat: add a schema for machine-readable provider conformance reports

    e483f47

    feat: carry results in Cucumber Messages instead of defining our own

    3560761

    feat(tck): emit a machine-readable conformance report

    fc6865c

    feat(tck): emit a machine-readable conformance report

    6878df8

    feat(tck): emit a machine-readable conformance report

    1c0f1a6

    feat: add a schema for machine-readable provider conformance reports

    40ae237

    feat: carry results in Cucumber Messages instead of defining our own

    6724bd3

    feat(tck): emit a machine-readable conformance report

    49d0fc2

    feat(tck): emit a machine-readable conformance report

    030ba5a

    feat(tck): emit a machine-readable conformance report

    b7dc8f7

    feat(tck): emit a machine-readable conformance report

    04e873d

    feat(tck): emit a machine-readable conformance report

    c7faa26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestionFurther information is requestedroadmap-proposalThis initiative is suggested for the project roadmap

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions