Repository navigation
feat(registry): capability registry for Test Management - #441
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
ruturaj-browserstack
left a comment
There was a problem hiding this comment.
Well-structured PR. The disabled-capability filter is applied once at load, errors come back by name, the grouped-arguments design avoids the parameter-name collisions, and the tests are thorough.
I ran the build gate locally on 3d54cd3: tsc, eslint and prettier are clean, vitest passes 1091/1091, and check:contract reports no new disagreements.
Requesting changes for:
- Path traversal in
bind(). A path parameter of..retargets a write:update_test_casewithtest_case_id: ".."becomesPATCH /api/v2/projects/PR-1/. Small fix plus a test. - CodeQL is still red. Alerts 6 and 7 are open in
seed-live.mts, and--showprints the full account.
Needs a team decision:
3. Tool descriptions total about 4k characters against the 120/200 budget in tool-design.md. Either trim them or record an exception.
4. Free-text telemetry (search_query, change_summary) needs the length cap that redact() claims to have, plus privacy sign-off.
Smaller items are inline. Nit: the PR body says 198 of 244, but the head commit and the index have 197.
Design note, not blocking: write consent rests on the model setting user_permission: "granted". The code says openly that this is never authorisation, and it's the same trust model as askBrowserStackAI. Worth the team agreeing on it explicitly now that 90+ write capabilities depend on it.
There was a problem hiding this comment.
[Low] Number("") is 0 (so is Number(" ")), so an empty ID becomes 0. Math.trunc silently turns "1.9" into 1. Suggest rejecting blank strings, and non-integers for integer parameters, instead of coercing them.
…, telemetry reasons Review findings from @ruturaj-browserstack and @SavioBS629 on #441. **A path value could retarget the request (High).** `encodeURIComponent` does not escape a dot and `new URL()` resolves `.` and `..`, so a string path parameter carrying `..` walked up a segment: update_test_case { project_id: "PR-1", test_case_id: ".." } -> PATCH /api/v2/projects/PR-1/ (the PROJECT, not a test case) Reproduced before fixing. Worse than a failed call: the human had approved "update this test case" and given a change_summary saying so, and the request edits a different resource with that approval already banked. Eleven enabled capabilities take a string path parameter with no pattern, enum or format, so nothing stopped it — the old comment claiming type checking was the defence held only for integer ids, and now says so. `""`, `"."` and `".."` are refused; a dot INSIDE a value ("v1.2.3") still binds. **Blank and fractional numbers were coerced (Low).** `Number("")` is 0, so a blank id became id 0 and the call went wherever that resolved. `Math.trunc` turned "1.9" into 1 silently. An id is either exact or it is the wrong id. **Credentials could go out over plaintext (Medium).** The host comes from an env var, account discovery or the index's own base_url, and nothing checked the scheme before the access key was attached. Now refused unless https, or a local host for development. **Redaction had no cap, and two comments said otherwise (Medium, privacy).** MAX_LENGTH is back at 512 and applied AFTER the rules, so a match cannot be split across the boundary. redact.ts claimed a cap it had lost; MCPEventExtras claimed "nothing here is free text written by a user" while `search_query` and `change_summary` are exactly that. Both corrected to say what actually happens. **Telemetry reason codes did not match the messages (Low).** `no_base_url` matched "no base URL is configured" while config.ts throws "no host is configured", so it never fired once. `capability_disabled` and `unknown_endpoint` fell through to the generic bucket — the first is the likeliest refusal on this build, with 46 of 244 capabilities withheld. Both InvocationError paths now carry the capability identity, so a failure row can be attributed. Every branch is pinned by a test against the message its source actually produces. **CodeQL alerts 4-7.** The first fix masked the account; CodeQL kept the alert open, correctly — it follows the value from process.env through any transform, and a mask is still a derived value on a path that did not need to carry it. The account is now omitted from every stdout path, including `--show`, which printed the pool file verbatim. The gitignored pool file still holds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A generic five-tool MCP surface — listProducts, describeEntity, searchCapability, describeCapability, invokeCapability — over a prebuilt per-product index, replacing one hand-written tool per endpoint. tm ships 244 capabilities, of which 198 are published. The other 46 are withheld behind a `disabled` flag: everything that did not pass live probing, plus the destructive tier. A disabled capability is filtered out when the registry loads, so search, describe and the entity listings cannot surface one and no reader downstream has to remember to check. Capability names carry no version suffixes. Where dropping `_v1`/`_v2` collided, the variants differ by ID FORM rather than API version, so the prefixed-identifier variant keeps the plain name and the integer one takes `_by_integer_id`. tests/live/CAPABILITY-RENAME-MAP.csv maps all 244 old names to new. The index was verified against the live API rather than against the spec it came from: 244 capabilities probed on preprod and two production accounts, with the run records, payloads and findings under tests/live/. Squashed onto main as a single commit. The development history is not carried over — it contained real user email addresses in probe fixtures and captured responses, and the working tree is clean of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(registry): record what was invoked, how the product answered, and why a call was refused Registry telemetry recorded only a tool name at entry, and every error path returned through `failed()` without recording anything. - `trackMCP` takes an optional `extras` object (scalars only), merged into the row. Existing callers are unaffected. - invokeCapability and describeCapability record the capability name, method, path, mode and product once resolved; invokeCapability adds `upstream_ok` and `upstream_status` when the product answers. `invoke()` returns 4xx/5xx rather than throwing, so `success` keeps its tool-level meaning and those two fields carry whether the call worked. - searchCapability records result count, total matched, truncated, weak_match and coverage, plus the search text. - A `refuse()` helper records and returns in one step, so every refusal carries a reason (unknown_capability, missing_parameter, bad_parameter_type, destructive_blocked, permission_not_granted, change_summary_missing, product_ambiguous, no_base_url). Refusals that never reach the network now produce a row. - Free text (`search_query`, `change_summary`) passes through `redact()` first: emails, URL credentials, JWTs, opaque tokens, IPs, phone numbers and long digit runs are replaced by shape, with no word lists, then capped at 200 characters. It fails closed, dropping the field rather than sending raw text. Names have no shape and are not covered; the module documents this. Registry rows stay identifiable by their tool names, and no field changes meaning for any existing tool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(registry): one telemetry row per invoke, not two invokeCapability recorded once after resolving the capability and again once the product answered, so a completed call produced two rows and counting invokes double-counted. The first record existed to keep the capability name if the call never returned, but every path between the two points already writes its own row through `refuse()`, so it was redundant on every path. Removing it leaves one row per invoke, carrying both the identity and the upstream result. A call killed mid-flight now produces no row rather than a partial one, which is the trade for a count that needs no filter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(registry): drop the length cap on recorded intent text Redaction still runs on the full string, so truncation was removing detail without adding protection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(registry): use the capability names the tm index publishes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot download Three corrections from live probing, all of them things the contract stated confidently and wrongly. `create_filter.filters` said only that its inner shape "is the same filter object the entity search takes". It is not. The UI applies a saved view only when its conditions are in the product's own filter-state shape — nested `systemFields` / `extraSystemFields` / `matchAllMatchAny`, where each selected option carries the project's numeric option id as `value` and its display name as `label`. Nothing validates it, so the wrong shape saves and returns success, then silently never applies. `extraSystemFields.issue_type` is required even for a view that does not filter on linked issues, for the same reason: omit it and the view saves but does nothing. `list_filters.entity_type` was published as "optional, despite being published as required", on the belief that omitting it returned one mixed list. It does not — omitting it is a 400. And its rows carry only `id`, `name` and `is_project_level`; the contract promised owner, entity_type, filters, project_id and timestamps that never arrive. `download_report` is withheld. It answers with a job channel this surface cannot resolve, so it can never deliver the file it is named for — the same test the other 46 were held to. tm now publishes 197 of 244. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL flagged two clear-text logs of a process-environment value (code-scanning alerts 4 and 5), and both are right. The startup line printed USERNAME directly, and the closing pool dump printed it again via pool.account. The username is not a credential by itself, but it is half of the `Api-Token: <username>:<access_key>` pair this surface authenticates with, and stdout here reaches CI logs, terminal scrollback and screen shares — read by more people than the shell that set the variable. The access key was never logged; that part was already right. Both sites now print a masked form (`ing…Xf`), which still distinguishes two accounts, which is all the caller needed it for. The pool FILE keeps the full value: it is gitignored, and knowing which account seeded a fixture is exactly what you want when a probe result looks wrong. Checked the sibling scripts for the same shape — replay-live.mts reads the same variables but never logs them, and nothing else in scripts/ or src/ interpolates a credential into a log line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, telemetry reasons Review findings from @ruturaj-browserstack and @SavioBS629 on #441. **A path value could retarget the request (High).** `encodeURIComponent` does not escape a dot and `new URL()` resolves `.` and `..`, so a string path parameter carrying `..` walked up a segment: update_test_case { project_id: "PR-1", test_case_id: ".." } -> PATCH /api/v2/projects/PR-1/ (the PROJECT, not a test case) Reproduced before fixing. Worse than a failed call: the human had approved "update this test case" and given a change_summary saying so, and the request edits a different resource with that approval already banked. Eleven enabled capabilities take a string path parameter with no pattern, enum or format, so nothing stopped it — the old comment claiming type checking was the defence held only for integer ids, and now says so. `""`, `"."` and `".."` are refused; a dot INSIDE a value ("v1.2.3") still binds. **Blank and fractional numbers were coerced (Low).** `Number("")` is 0, so a blank id became id 0 and the call went wherever that resolved. `Math.trunc` turned "1.9" into 1 silently. An id is either exact or it is the wrong id. **Credentials could go out over plaintext (Medium).** The host comes from an env var, account discovery or the index's own base_url, and nothing checked the scheme before the access key was attached. Now refused unless https, or a local host for development. **Redaction had no cap, and two comments said otherwise (Medium, privacy).** MAX_LENGTH is back at 512 and applied AFTER the rules, so a match cannot be split across the boundary. redact.ts claimed a cap it had lost; MCPEventExtras claimed "nothing here is free text written by a user" while `search_query` and `change_summary` are exactly that. Both corrected to say what actually happens. **Telemetry reason codes did not match the messages (Low).** `no_base_url` matched "no base URL is configured" while config.ts throws "no host is configured", so it never fired once. `capability_disabled` and `unknown_endpoint` fell through to the generic bucket — the first is the likeliest refusal on this build, with 46 of 244 capabilities withheld. Both InvocationError paths now carry the capability identity, so a failure row can be attributed. Every branch is pinned by a test against the message its source actually produces. **CodeQL alerts 4-7.** The first fix masked the account; CodeQL kept the alert open, correctly — it follows the value from process.env through any transform, and a mask is still a derived value on a path that did not need to carry it. The account is now omitted from every stdout path, including `--show`, which printed the pool file verbatim. The gitignored pool file still holds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edit_test_run replaces whatever it is not sent, and the only thing that
knows what to send back is GET .../test-runs/{id}/edit/selection — the
read the product UI makes before every "Add to Run" edit. It was missing
from the index, so an agent could not use the full edit safely.
Add get_test_run_edit_selection (read, test_run, not permission-gated)
from teststack test_runs#get_selection, with a TestRunEditSelectionResponse
schema, and point edit_test_run at it: call it first, resend selection,
shared_selection, trtc_configuration_mappings and every run field.
Checked live on an open run (TR-5448072) and a closed one (TR-5448258).
Two things the source did not show went into the guidance:
- a run with no configurations still returns one default
trtc_configuration_mappings entry (configuration_id null, select_all true);
- the stored selection is not kept in step with the add-only and remove
calls — TR-5448258 still listed a removed case and a folder stored as
select-all (unique_test_case_count 10 vs 9 cases). Resending it as-is
would re-add cases, so the guidance says to compare counts first.
Published count 197 -> 198 of 245.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
close_test_run's guidance said a run could be reopened by editing its run_state back, and update_latest_test_result advised reopening a closed run. Neither is true. Tried on TR-5448258 after closing it: - update_test_run (v2 PATCH, run_state in_progress) -> 404 "Test Run ID is invalid", because the builds-backed lookup does not resolve closed runs. - edit_test_run (v1 full edit, every field resent) -> 200 success, run still closed, closed_at unchanged. Backend: testhub's run update throws "Closed Testruns can not be modified" for any closed run (use_cases/testRun.js). The rearch builds path (teststack builds_public_update_testrun_th -> testhub updateBuildsByUuid) has no such guard and returns success instead — reported separately. close_test_run now says closing is final and points to cloning the run to keep testing; run_state on update_test_run, edit_test_run and replace_test_run notes that closed is one-way and how each path answers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…fixed list `loads the stored layout` pinned listProducts to ["tm"], so it broke the moment a second index landed beside it in the fixture directory — which is the case the stored layout exists for. One product could never prove that one-file-per-product discovery works. The expectation now reads the directory, so the test asserts what discovery found rather than what it was written next to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e28a886 to
7d7ac08
Compare

Replaces one hand-written MCP tool per endpoint with a generic five-tool surface over a prebuilt, per-product index.
listProducts·describeEntity·searchCapability·describeCapability·invokeCapabilityWhat ships
198 of 244 tm capabilities. The other 46 are withheld behind a
disabledflag while the product team works through them: everything that did not pass live probing, plus the destructive tier.The flag is enforced structurally, not by convention. Disabled capabilities are filtered out when the registry loads, so
searchCapability,describeCapabilityand the entity listings cannot surface one, and no reader downstream has to remember to check. There are four readers of the capabilities array plus the entity docs, and the cost of missing one is a capability hidden from search and still invokable — precisely the failure the flag exists to prevent.invokeCapabilityis the one exception: an exact-handle invoke answerscapability_disabledrather thanunknown_capability, so a caller that already had the name learns the absence is deliberate instead of hunting for a synonym.No destructive capability is published at all — deletes are withheld from the surface rather than refused after the fact, so they never appear in a search result.
Verified against the live API, not against the spec
Every capability was probed against preprod and two production accounts. That is the only check that catches the index lying — promising a body shape the API rejects, or a response field it never returns. It found capabilities that had shipped broken and unusable since the index was created, which no structural test could have seen.
scripts/check-contract.pyruns in CI (npm run check:contract) and fails the build when a capability'sreturnsdisagrees with its resolved 2xx schema. It compares againstscripts/contract-baseline/tm.json, so it breaks on new disagreements rather than on the known set.Naming
No version suffixes. Where dropping
_v1/_v2collided, the variants differ by ID form rather than API version — one takes an integerproject_id, the other a prefixedPR-NNN— so the prefixed variant keeps the plain name and the integer one takes_by_integer_id, matching the convention already in the index.Five renames go further and correct a name that described the wrong behaviour. The worst promised an undo and performed a recursive delete.
Reviewing this
92% of the diff is two generated JSON files. The reviewable surface is ~7,000 lines:
capability/— the generated indextests/fixtures/— test datasrc/— registry codetests/— test codescripts/— toolingtests/fixtures/capability/tm.capability-index.json(41,894 lines) is a frozen copy the tests pin against — safe to skip.tests/fixtures/capability-multi/holds a synthetic second product: a made-up widget catalogue with dummy capabilities, never published. The cross-product routing gate is shipped code, so it needs two products under test even though only tm ships. Its README records the two non-obvious constraints the tests place on it.Notes
main.🤖 Generated with Claude Code