Conversation
… phase 1 Adds the requirement-traceability yaml for MCP Events plus the first two of five server scenarios, scoring against the design sketch that merged on main of modelcontextprotocol/experimental-ext-triggers-events on 2026-09-08. The SEP number is a placeholder. Events has no PR in modelcontextprotocol/modelcontextprotocol, and both traceability gates are numeric (the filename regex at src/traceability/index.ts:174, the check-id regex at line 41), so an unnumbered file is dropped from the manifest without an error. 9999 is far from the live range; the rename is mechanical and the yaml header names the rename as a merge blocker rather than a follow-up: a merge under 9999 would publish it as a real SEP. The reservation question is open with the WG. src/seps/sep-9999.yaml declares 131 checks and 30 excluded rows against 144 RFC 2119 keyword occurrences. Pure MAY and OPTIONAL sentences get no check. Client, host, receiver and SDK-guidance obligations are excluded rather than declared, so the denominator holds only what a server-side run can observe. events-discovery covers the capability, events/list, the descriptor fields and the error-code contract. events-poll covers poll delivery, the EventOccurrence shape and cursor lifecycle, driving a real two-poll quiet-period loop so cursor advancement is gradeable against a server with no traffic. Together they emit 45 rows; the other 86 report untested until the push and webhook scenarios land. The capability gate asks before it skips. An optional capability a server never declared is not a defect, but a server that answers events/list while declaring nothing has a surface no spec-following client would reach, and a SKIP would report that as a clean run. mcpkit is in exactly that state. Against examples/events/kitchen-sink at mcpkit main: discovery 8/11, poll 18/28. Six divergences, five of them server-side defects not previously tracked, one confirming the known nextPollSeconds drift. The yaml header lists each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
commit: |
There was a problem hiding this comment.
Flagging this deliberately, because it is the one place this suite departs from how every other extension scenario in the repo behaves, and the departure is what turns the run against mcpkit red rather than green.
Everywhere else here, an optional capability the server never declared is a clean SKIP. That is correct when the server genuinely does not implement the feature. It is wrong when the server implements it and never says so: mcpkit answers events/list, serves all three delivery modes, and declares nothing under capabilities.events, so a client that reads capabilities to decide what to call never reaches any of it. A SKIP reports that as a clean run over a surface no client can use.
So the gate asks before it skips. One extra events/list separates "does not do events" (-32601, skip everything) from "does events off-menu" (returns a catalog, fail the declaration check and keep grading the rest).
The precedent is declaredSkillsCapability in src/scenarios/server/skills/helpers.ts, split out from skillsCapability for the same reason: a malformed declaration was being folded into "not declared" and skipping the whole suite, which "reads as a clean run against a server that is plainly wrong". This applies that argument to an absent declaration rather than a malformed one.
The cost is one extra request against servers that do not implement events at all. If you would rather pay nothing there and accept the false SKIP, say so, but the finding seems worth the request.
Adds the testconf-events target, its runner-agnostic script, and the local-suites manifest entry, driving examples/events/kitchen-sink against the two phase-1 scenarios (events-discovery, events-poll). Those are proposed upstream as a draft in modelcontextprotocol/conformance#504 and tracked on the panyam/mcpconformance fork branch until it lands. INFO rather than a gate. The suite scores against the Events design sketch that merged upstream on 2026-09-08, which has no SEP number yet, so its check IDs still carry a placeholder sep-9999- prefix. And the run is red on purpose: the failures are our divergences from that document, which is what the suite was built to surface. 27 pass / 13 fail / 5 warn against kitchen-sink at c9160cf. Five divergences, all tracked in #1380: events/list answers while no capabilities.events is declared, so the whole surface is unreachable for a client that reads capabilities first; events/poll omits the events key instead of returning an empty array; and the events.topology meta-source does not keep the descriptor contract, sending delivery null, carrying no inputSchema, and answering events/poll despite advertising no poll delivery. #1379 closed the nextPollSeconds rename and #1381 added list_changed and termination plus inputSchema on the three real sources, so those no longer show up here. Flip this to a gate once #1380 closes and the spec text stabilises. path-defaults.{mk,sh,just} are regenerated via gen_conf_paths.py rather than hand-edited. check_local_suites.py reports no drift across 9 suites. Refs #1374 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
Adds the testconf-events target, its runner-agnostic script, and the local-suites manifest entry, driving examples/events/kitchen-sink against the two phase-1 scenarios (events-discovery, events-poll). Those are proposed upstream as a draft in modelcontextprotocol/conformance#504 and tracked on the panyam/mcpconformance fork branch until it lands. INFO rather than a gate. The suite scores against the Events design sketch that merged upstream on 2026-09-08, which has no SEP number yet, so its check IDs still carry a placeholder sep-9999- prefix. And the run is red on purpose: the failures are our divergences from that document, which is what the suite was built to surface. 27 pass / 13 fail / 5 warn against kitchen-sink at c9160cf. Five divergences, all tracked in #1380: events/list answers while no capabilities.events is declared, so the whole surface is unreachable for a client that reads capabilities first; events/poll omits the events key instead of returning an empty array; and the events.topology meta-source does not keep the descriptor contract, sending delivery null, carrying no inputSchema, and answering events/poll despite advertising no poll delivery. #1379 closed the nextPollSeconds rename and #1381 added list_changed and termination plus inputSchema on the three real sources, so those no longer show up here. Flip this to a gate once #1380 closes and the spec text stabilises. path-defaults.{mk,sh,just} are regenerated via gen_conf_paths.py rather than hand-edited. check_local_suites.py reports no drift across 9 suites. Refs #1374 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL

What changes
Adds a requirement-traceability yaml for MCP Events and the first two of five server scenarios, scoring an implementation against the design sketch that merged on
mainofmodelcontextprotocol/experimental-ext-triggers-eventson 2026-09-08.events-discoverygrades the capability declaration,events/listand the error-code contract.events-pollgrades poll delivery, theEventOccurrenceshape and cursor lifecycle. Together they emit 45 of the 131 declared rows; push and webhook follow in later phases and reportuntesteduntil then.Provenance
panyam offered an Events conformance suite in
#triggers-events-wgon 2026-08-28 and Peter Alexander accepted on 2026-08-31 ("No objections, that would be great!"). This is the first installment. It follows the SEP-2640 suite (#330) in shape: extract the requirements into a traceability yaml first, then write scenarios that emit one check per row.Per
AGENTS.mdthis would normally start as an issue. The WG conversation is that discussion, and the extraction is easier to argue about as a diff than as prose, so it is here as a draft instead. Happy to split it into an issue if you would rather.Prerequisite knowledge
events-polltests; Subscribing and Event Delivery is the densest normative section and is mostly still ahead of us.AGENTS.md§ Scenario design and § Check conventions — why this is two scenarios with 45 checks rather than 45 scenarios, and why a missing prerequisite fails rather than skips (Standardize how scenarios report setup/execution failures #248).src/traceability/— how a declared row becomestestedoruntested. This is what makes declaring all 131 rows up front the right move rather than an overreach.src/seps/sep-2640.yaml— the closest precedent for the yaml's shape and for how much provenance belongs in its header.The number question
This is the part I would most like a decision on, because it is the only thing blocking merge.
SEP numbers are PR numbers in
modelcontextprotocol/modelcontextprotocol, and Events has no PR there — the work lives in its own repository. Both traceability gates are numeric: the directory filter atsrc/traceability/index.ts:174(/^sep-\d+\.yaml$/) and the check-id filter at line 41 (/^sep-\d+-/). A file namedevents.yamlis silently dropped from the manifest rather than rejected, andsrc/new-sep/index.ts:163refuses any argument that is not a positive integer. So there is no honest name that works today.Three ways out, and I have no strong preference between the first two:
I used
9999to unblock the extraction, deliberately far from the live range. The rename touches the filename, thesep:field, everysep-9999-id in the yaml, and the same ids undersrc/scenarios/server/events/— mechanical, but wide enough that it is worth doing once, in the right direction.Reviewer's guide
A restaurant can cook a dish that is not on the menu. The kitchen has the ingredients, the line knows the recipe, and if you somehow know to ask you get it — but nobody reading the menu ever orders it, so from the outside it may as well not exist. That is the state this suite found its first implementation in:
events/listanswers, all three delivery modes work, and nothing appears undercapabilities.events, which is the menu a spec-following client reads before deciding what to ask for. This matters for how the suite is shaped, because the obvious design misses it. A conformance harness normally treats an undeclared optional capability as not-applicable and skips, which is right for a restaurant that genuinely does not serve the dish and wrong for one that serves it off-menu: the skip reports a clean run over a surface no client can reach. So the capability gate here asks before it skips — it callsevents/list, and only a server that both declares nothing and implements nothing gets the skip. The same instinct runs through the rest of the diff. Wherever a check cannot be exercised, the scenario names the missing prerequisite and fails, because the one thing worse than a red conformance run is a green one that tested nothing.Read in this order:
EVENTS_EXTENSION_IDis only a suite-selection key. Events declares its capability top-level, not undercapabilities.extensions, unlike every other extension here, so that constant is never a path into the capability object.checks(). Everything after it grades whatever the server actually serves, field by field.quietAdvanceChecks: two polls with the first response's cursor fed into the second, which is what makes cursor advancement gradeable against a server where nothing is happening. That is the state a conformance fixture is usually in.How it works
The
truncatedrows that need a stale cursor are reported untestable rather than skipped. Cursors are opaque by definition, so the harness cannot mint one old enough to fall outside a retention window, and saying so is more useful than a silent pass.Decision log
untested, which is the manifest working rather than a gap in this file.MAYandOPTIONALsentences get no row. Five initially got one and were demoted toexcluded:rather than deleted, so the keyword sweep stays auditable against the source.EventOccurrencefield table, the error-code table and the descriptor fields, which the sketch states declaratively. They are normative and directly testable. Since "severity follows the keyword" is undefined for them, the yaml header states the rule used: a field the document marks required, or a wire fact a client cannot work without, isFAILURE; anything else drawn from an example payload isWARNING.https— which localhost cannot satisfy, since the same SSRF rules the suite checks require a conformant server to refuse it.src/scenarios/server/skills/negative.test.ts. An events-capable fixture inexamples/servers/typescript/is real work and belongs with the push and webhook scenarios, which genuinely need one.new-sep's PR lookup does not apply andspecPathToUrl(src/new-sep/index.ts:19) hard-requires adocs/specification/draft/prefix that does not exist here. Scaffolded with--spec-url. There are no section anchors to cite, so rows carry no per-rowurlandspec_urlnames the document itself.Risk / blast radius
EXTENSION_IDS. No existing scenario, suite list or check id changes.sourceis{ extensionId }, somatchesSpecVersionreturns false for every--spec-versionand these never join a dated run. They are also in the pending list. Reaching them takes--scenario events-*or--suite all.src/scenarios/server/events/negative.test.ts, 29 cases. Full suite 651 pass across 48 files.src/seps/traceability.jsonwill drift once these ids are emitted in a real run. PerAGENTS.mdthe traceability workflow refreshes it by PR and it is not a gate, so this leaves it alone.Before / after
There was no Events suite before, so what matters is what an implementation looks like when scored. Run against mcpkit's
examples/events/kitchen-sinkat itsmain:Six divergences, each observed against the running server rather than inferred from source:
sep-9999-capability-events-objectevents/listwhile declaring nocapabilities.eventssep-9999-descriptor-input-schemainputSchemasep-9999-descriptor-delivery-subsetevents.topologysendsdelivery: nullwhere an array is requiredsep-9999-poll-events-arrayeventsinstead of returning an empty arraysep-9999-poll-mode-unsupportedevents/pollfor a type advertising onlypush/webhooksep-9999-poll-next-poll-msnextPollSeconds, the name spec commit197c32b4retiredFive of the six were not previously tracked by that implementation. They are here as evidence the checks catch something rather than as the point of the PR — the SEP-2640 experience was that running a suite against an implementation nobody wrote it for is what finds the suite's own bugs, and a second implementation is next.
flowchart TD A[server under test] --> B{capabilities.events declared?} B -->|yes| G[grade everything] B -->|no| C{events/list answers?} C -->|"-32601"| D[SKIP all: does not do events] C -->|returns a catalog| E[FAIL the declaration check] E --> G G --> H{check exercisable?} H -->|yes| I[SUCCESS or FAILURE] H -->|no| J[untestable: FAIL and name the prerequisite]Out of scope
events-push,events-webhook,events-webhook-delivery— later phases. Their 86 rows are declared and reportuntested.--headeroption on theserverrunner. Needed to score a second implementation that sits behind OAuth, which the runner currently cannot reach at all. Its own PR.sep-9999-*once the number question above is settled. Merge blocker.