feat(tck): emit a machine-readable conformance report · open-feature/python-sdk-contrib@04e873d · GitHub
Skip to content

Commit 04e873d

Browse files
aepfliaepfli
committed
feat(tck): emit a machine-readable conformance report
Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Unset means no report, which is the default and is not an error. An environment variable rather than a TckConfig field, so that emitting a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. Several suites in one pytest session each write their own file, so flagd's two resolvers would not collide. The load-bearing part is the per-scenario list. Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and nothing downstream can check that against a summary line. Recording every scenario's outcome individually makes the rule checkable by the consumer instead of dependent on the runner. It is also required to be complete, because a document that quietly dropped what it skipped would satisfy the letter of the rule and still mislead whoever read it. pytest, unlike godog, reports a skip honestly -- so the interesting divergence here is elsewhere. The one scenario the Python SDK cannot satisfy is marked xfail, so the run finishes green; the provider still did not satisfy it, and the document says failed with the reason. An expected failure is a recorded deviation, not an excused one. Scenarios are therefore enumerated at collection and resolved at the end of the session rather than as fixtures run, which is also what keeps a scenario skipped by a marker -- whose fixtures never run at all -- from vanishing from the document. Identity comes from spec_revision.json, generated by hatch_build_sync.py beside the copied assets and force-included into the wheel. It has to be captured at build time: the submodule that knows the answer is not in the distribution, so an installed copy has nothing left to ask. A build that cannot reach git -- an unpacked sdist -- warns and records "unknown" rather than inventing a commit. Both the commit and the tree hash are recorded, the tree because it identifies the assets alone: unchanged by unrelated edits elsewhere in the specification, so two runs of identical assets agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with TckConfig.name recorded as the configuration, because TckConfig.name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read off an optional control_api property rather than added to the BackendControl protocol, so that adding it leaves every existing control complete and one that stays quiet simply omits the field. The tests assert the two properties a consumer is entitled to assume -- that no scenario the capability gate stopped is ever reported as passed, and that every collected scenario appears exactly once, counted against pytest's own collection rather than against a number written down beside it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
1 parent 06cb43f commit 04e873d

12 files changed

Lines changed: 1452 additions & 49 deletions

File tree

‎tools/openfeature-tck/.gitignore‎

Lines changed: 3 additions & 0 deletions

‎tools/openfeature-tck/hatch_build.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,15 +18,28 @@
1818
# the single definition of what gets copied where -- would not be importable.
1919
sys.path.insert(0, str(Path(__file__).parent))
2020

21-
from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync
21+
from hatch_build_sync import (
22+
FILES,
23+
PACKAGE_REL,
24+
REVISION_FILE,
25+
SPEC_ASSETS,
26+
TREES,
27+
sync,
28+
)
2229

2330

2431
class SpecAssetsCopyHook(BuildHookInterface):
2532
PLUGIN_NAME = "spec-assets-copy"
2633

2734
def initialize(self, version: str, build_data: dict) -> None:
2835
root = Path(self.root)
29-
copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES]
36+
# The generated revision file travels with the assets it describes. It
37+
# has to be built here rather than read at run time, because the
38+
# submodule that knows the answer is not in the wheel and a conformance
39+
# report has to name the revision it ran against.
40+
copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + [
41+
root / PACKAGE_REL / REVISION_FILE
42+
]
3043

3144
# Building from a checkout: refresh from the submodule, so what ships is
3245
# always the revision the pin names. Building from an sdist: there is no

‎tools/openfeature-tck/hatch_build_sync.py‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515

1616
from __future__ import annotations
1717

18+
import json
1819
import os
1920
import shutil
2021
import subprocess
@@ -25,7 +26,8 @@
2526
ROOT = Path(__file__).parent
2627
SPEC_DIRNAME = "spec"
2728
SPEC_ROOT = (ROOT / SPEC_DIRNAME).resolve()
28-
SPEC_ASSETS = (SPEC_ROOT / "specification/assets/provider-tck").resolve()
29+
ASSETS_PATH_IN_SPEC = "specification/assets/provider-tck"
30+
SPEC_ASSETS = (SPEC_ROOT / ASSETS_PATH_IN_SPEC).resolve()
2931
PACKAGE_REL = Path("src/openfeature/contrib/tools/tck")
3032
DEST_BASE = ROOT / PACKAGE_REL
3133

@@ -243,6 +245,28 @@ def checkout_pinned_spec(git: GitRunner = _run_git) -> str | None:
243245
return pinned
244246

245247

248+
REVISION_FILE = "spec_revision.json"
249+
"""Which revision of the specification the copied assets came from.
250+
251+
Recorded at build time because the answer is only available at build time: the
252+
submodule that holds it is not in the wheel, and a conformance report that cannot
253+
name the revision it ran against cannot be compared with another. It is generated
254+
by the same command that copies the assets, which is what keeps the two from
255+
disagreeing.
256+
257+
Not committed, for the same reason the assets are not: the submodule pin is the
258+
single record of which revision this package targets.
259+
"""
260+
261+
UNKNOWN_REVISION = "unknown"
262+
"""Seven characters, the minimum the report schema accepts.
263+
264+
A build that cannot reach git says it does not know rather than inventing a
265+
commit, and still produces a document that validates. Which happens for real:
266+
building from a source tarball has no ``.git`` to ask.
267+
"""
268+
269+
246270
def sync() -> None:
247271
checkout_pinned_spec()
248272

@@ -267,6 +291,50 @@ def sync() -> None:
267291
dest.unlink()
268292
shutil.copy2(SPEC_ASSETS / src_name, dest)
269293

294+
write_revision()
295+
296+
297+
def write_revision() -> None:
298+
"""Record the spec commit and the asset tree these copies came from.
299+
300+
The tree hash is carried as well as the commit because it identifies the
301+
assets alone: it does not change when an unrelated part of the specification
302+
does, so two runs that executed identical assets report the same value even
303+
when pinned to different commits. It is also checkable rather than merely
304+
asserted, since ``git rev-parse <commit>:specification/assets/provider-tck``
305+
must reproduce it.
306+
"""
307+
commit = _git("rev-parse", "HEAD") or UNKNOWN_REVISION
308+
tree = _git("rev-parse", f"HEAD:{ASSETS_PATH_IN_SPEC}") or ""
309+
(DEST_BASE / REVISION_FILE).write_text(
310+
json.dumps({"specRevision": commit, "assetsTree": tree}, indent=2) + "\n",
311+
encoding="utf-8",
312+
)
313+
314+
315+
def _git(*args: str) -> str:
316+
"""Run git inside the submodule, returning its output or an empty string.
317+
318+
A build must not hard-fail because git is absent or the checkout is not a
319+
repository -- both are ordinary when building from an unpacked sdist. The
320+
failure is reported as a warning and the identity degrades to ``unknown``,
321+
which is legible in the resulting report rather than silently wrong.
322+
"""
323+
command = ["git", "-C", str(SPEC_ROOT), *args]
324+
try:
325+
completed = subprocess.run( # noqa: S603
326+
command, capture_output=True, check=True, text=True
327+
)
328+
except (OSError, subprocess.CalledProcessError) as error:
329+
warnings.warn(
330+
f"could not determine the spec revision ({' '.join(command)}: {error}); "
331+
f"conformance reports from this build will not name the revision they "
332+
f"ran against",
333+
stacklevel=2,
334+
)
335+
return ""
336+
return completed.stdout.strip()
337+
270338

271339
if __name__ == "__main__":
272340
sync()

‎tools/openfeature-tck/pyproject.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ artifacts = [
6969
"src/openfeature/contrib/tools/tck/gherkin/",
7070
"src/openfeature/contrib/tools/tck/flag_data/",
7171
"src/openfeature/contrib/tools/tck/control-api.yaml",
72+
# Which spec revision those assets came from, generated beside them. The
73+
# submodule is not in the wheel, so a conformance report emitted by an
74+
# installed copy has no other way to name the revision it ran against.
75+
"src/openfeature/contrib/tools/tck/spec_revision.json",
7276
]
7377

7478
[tool.hatch.build.hooks.custom]

‎tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py‎

Lines changed: 4 additions & 0 deletions

0 commit comments

Comments
 (0)