test(e2e): checks.go as the reviewed vocabulary, SCENARIO.md codifies… · docker/compose@c1028c2 · GitHub
Skip to content

Commit c1028c2

Browse files
ndeloofclaude
andcommitted
test(e2e): checks.go as the reviewed vocabulary, SCENARIO.md codifies the contract
Checks are the shared vocabulary between scenarios: move them to a dedicated checks.go so the whole vocabulary is read — and reviewed — as one file, independently of the tests using it. SCENARIO.md codifies the rules that so far only lived in the PR description: one intent = one invariant, step names as behavior sentences, inline compose model, state-based checks first with OutputContains as an explicit last resort, new checks generic and named after their observable — plus how to exploit the failure artifacts and E2E_KEEP_FAILED. AGENTS.md points new e2e work at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
1 parent 3b7c652 commit c1028c2

4 files changed

Lines changed: 340 additions & 210 deletions

File tree

‎AGENTS.md‎

Lines changed: 9 additions & 0 deletions

‎pkg/e2e/SCENARIO.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# E2E scenarios: the contract
2+
3+
With coding agents writing most of the production code, e2e tests are the
4+
document humans actually read and review. A scenario must state an intent, a
5+
compose model, and a sequence of steps `(command → expected observables)` —
6+
and nothing else. Everything operational (project naming, cleanup, failure
7+
diagnostics) belongs to the framework, not to the test.
8+
9+
The DSL lives in [`scenario.go`](scenario.go) (execution, actions,
10+
requirements) and [`checks.go`](checks.go) (the vocabulary of observables).
11+
12+
## Writing a scenario
13+
14+
```go
15+
func TestRestart(t *testing.T) {
16+
NewScenario(t, "restart must bring an exited service back up, restarting the same container").
17+
Compose(`
18+
services:
19+
app:
20+
image: alpine
21+
init: true
22+
command: ash -c "if [[ -f /tmp/restart.lock ]] ; then sleep infinity; else touch /tmp/restart.lock; fi"
23+
`).
24+
Step("up starts the service, whose first run exits at once",
25+
ComposeCmd("up", "-d"),
26+
Eventually(ServiceState("app", "exited"), 10*time.Second)).
27+
Step("restart brings the service back up, reusing the container",
28+
ComposeCmd("restart"),
29+
Eventually(ServiceState("app", "running"), 10*time.Second),
30+
NotRecreated("app"))
31+
}
32+
```
33+
34+
Rules:
35+
36+
- **New e2e tests use `NewScenario`.** The legacy `NewCLI` style remains for
37+
existing tests, converted opportunistically; don't add to it.
38+
- **One intent = one invariant.** The intent is a one-line statement of the
39+
behavior being locked, phrased as an obligation ("X must Y"). If you need
40+
two intents, write two scenarios.
41+
- **Step names are behavior sentences**, not command echoes: "an unchanged
42+
create is a no-op", not "run create again". The transcript of step names
43+
should read as the specification.
44+
- **The compose model is inline.** A scenario is self-contained in the test
45+
source; no fixture directories. Interpolate runtime values via `Env`.
46+
- **Regression tests link the issue** in a comment above the test, with a
47+
sentence on the failure mode being locked.
48+
49+
## Checks: observe real state
50+
51+
Checks are the shared vocabulary between scenarios; their discipline is what
52+
keeps the contract meaningful.
53+
54+
- **Prefer state-based checks** (`ServiceState`, `NotRecreated`, `LabelSet`,
55+
`RunsOnPlatform`, …): they observe containers, labels and image manifests —
56+
what the user actually gets — not what the CLI printed.
57+
- **`OutputContains` is a last resort**, legitimate only when the CLI's
58+
reported decision is itself the observable (e.g. "Skipped" vs "Pulled").
59+
- **Never poll by hand**: wrap a state check in `Eventually(check, timeout)`.
60+
No `time.Sleep` in scenarios.
61+
- **A new check must be generic** — no test-specific logic — **and named
62+
after the observable it asserts**, not after the test that needed it. It
63+
goes in `checks.go`, where the whole vocabulary is reviewed as one file.
64+
Before adding one, verify the observable isn't already expressible.
65+
- A check should also fail loudly on a broken precondition (e.g.
66+
`NotRecreated` errors if the service had no container before the step)
67+
rather than pass vacuously.
68+
69+
## When a scenario fails
70+
71+
The report opens with everything needed to diagnose without re-running:
72+
73+
- **`artifacts: <dir>`** — a stable per-project directory holding the
74+
untruncated material: `compose.yaml`, `failure.txt`, each step's full
75+
command and output (`step-NN-*.txt`), `containers.txt`, `events.txt`, full
76+
container logs (`logs-*.txt`) and the per-step state snapshots
77+
(`snapshots.json`). Read these before re-running anything.
78+
- **`E2E_KEEP_FAILED=1`** — rerun with this set to skip teardown of failed
79+
scenarios: containers, volumes and networks stay alive for `docker
80+
inspect`/`exec`. Clean up afterwards with
81+
`docker compose --project-name <project> down -v --remove-orphans`.
82+
- The inline report shows the transcript (every step, exit code, duration),
83+
the failing step's output, project containers, engine events since the
84+
scenario started, and container log tails — truncated for readability; the
85+
artifacts have the full versions.
86+
87+
Run a single scenario with:
88+
89+
```sh
90+
go test -tags e2e ./pkg/e2e/ -run TestRestart -v
91+
```

‎pkg/e2e/checks.go‎

Lines changed: 240 additions & 0 deletions

0 commit comments

Comments
 (0)