feat(registry): capability registry for Test Management by sarve-shreyas · Pull Request #441 · browserstack/mcp-server · GitHub
Skip to content

feat(registry): capability registry for Test Management - #441

Merged
gaurav-singh-9227 merged 8 commits into
mainfrom
feat/tm-capability-registry
Sep 28, 2026
Merged

gaurav-singh-9227 merged 8 commits into
mainfrom
feat/tm-capability-registry

Conversation

@sarve-shreyas

Copy link
Copy Markdown
Collaborator

Replaces one hand-written MCP tool per endpoint with a generic five-tool surface over a prebuilt, per-product index.

listProducts · describeEntity · searchCapability · describeCapability · invokeCapability

What ships

198 of 244 tm capabilities. The other 46 are withheld behind a disabled flag while the product team works through them: everything that did not pass live probing, plus the destructive tier.

The flag is enforced structurally, not by convention. Disabled capabilities are filtered out when the registry loads, so searchCapability, describeCapability and the entity listings cannot surface one, and no reader downstream has to remember to check. There are four readers of the capabilities array plus the entity docs, and the cost of missing one is a capability hidden from search and still invokable — precisely the failure the flag exists to prevent.

invokeCapability is the one exception: an exact-handle invoke answers capability_disabled rather than unknown_capability, so a caller that already had the name learns the absence is deliberate instead of hunting for a synonym.

No destructive capability is published at all — deletes are withheld from the surface rather than refused after the fact, so they never appear in a search result.

Verified against the live API, not against the spec

Every capability was probed against preprod and two production accounts. That is the only check that catches the index lying — promising a body shape the API rejects, or a response field it never returns. It found capabilities that had shipped broken and unusable since the index was created, which no structural test could have seen.

scripts/check-contract.py runs in CI (npm run check:contract) and fails the build when a capability's returns disagrees with its resolved 2xx schema. It compares against scripts/contract-baseline/tm.json, so it breaks on new disagreements rather than on the known set.

Naming

No version suffixes. Where dropping _v1/_v2 collided, the variants differ by ID form rather than API version — one takes an integer project_id, the other a prefixed PR-NNN — so the prefixed variant keeps the plain name and the integer one takes _by_integer_id, matching the convention already in the index.

Five renames go further and correct a name that described the wrong behaviour. The worst promised an undo and performed a recursive delete.

Reviewing this

92% of the diff is two generated JSON files. The reviewable surface is ~7,000 lines:

lines
capability/ — the generated index 50,300
tests/fixtures/ — test data 44,589
src/ — registry code 3,721
tests/ — test code 3,315
scripts/ — tooling 1,999
config (CI, package.json, README) 92

tests/fixtures/capability/tm.capability-index.json (41,894 lines) is a frozen copy the tests pin against — safe to skip.

tests/fixtures/capability-multi/ holds a synthetic second product: a made-up widget catalogue with dummy capabilities, never published. The cross-product routing gate is shipped code, so it needs two products under test even though only tm ships. Its README records the two non-obvious constraints the tests place on it.

Notes

  • Squashed to a single commit, rebased onto current main.
  • 1064 tests, typecheck, lint and the contract gate all clean.
  • README documents the five tools as 47–51 in the existing list.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited)

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 59e6b774-ec9a-46b4-afb5-340c20e5ddd9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Comment thread scripts/seed-live.mts Fixed
Comment thread scripts/seed-live.mts Fixed
@sarve-shreyas
sarve-shreyas added this pull request to stack #448 September 27, 2026 15:28
Comment thread scripts/seed-live.mts Fixed
Comment thread scripts/seed-live.mts Fixed

@ruturaj-browserstack ruturaj-browserstack left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-structured PR. The disabled-capability filter is applied once at load, errors come back by name, the grouped-arguments design avoids the parameter-name collisions, and the tests are thorough.

I ran the build gate locally on 3d54cd3: tsc, eslint and prettier are clean, vitest passes 1091/1091, and check:contract reports no new disagreements.

Requesting changes for:

  1. Path traversal in bind(). A path parameter of .. retargets a write: update_test_case with test_case_id: ".." becomes PATCH /api/v2/projects/PR-1/. Small fix plus a test.
  2. CodeQL is still red. Alerts 6 and 7 are open in seed-live.mts, and --show prints the full account.

Needs a team decision:
3. Tool descriptions total about 4k characters against the 120/200 budget in tool-design.md. Either trim them or record an exception.
4. Free-text telemetry (search_query, change_summary) needs the length cap that redact() claims to have, plus privacy sign-off.

Smaller items are inline. Nit: the PR body says 198 of 244, but the head commit and the index have 197.

Design note, not blocking: write consent rests on the model setting user_permission: "granted". The code says openly that this is never authorisation, and it's the same trust model as askBrowserStackAI. Worth the team agreeing on it explicitly now that 90+ write capabilities depend on it.

Comment thread src/tools/capability-registry/bind.ts Outdated
Comment thread src/tools/capability-registry/bind.ts Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Number("") is 0 (so is Number(" ")), so an empty ID becomes 0. Math.trunc silently turns "1.9" into 1. Suggest rejecting blank strings, and non-integers for integer parameters, instead of coercing them.

Comment thread src/tools/capability-registry/register.ts
Comment thread src/tools/capability-registry/register.ts Outdated
Comment thread src/tools/capability-registry/redact.ts Outdated
Comment thread src/tools/capability-registry/discovery.ts
Comment thread scripts/seed-live.mts Outdated
sarve-shreyas added a commit that referenced this pull request Sep 28, 2026
…, telemetry reasons

Review findings from @ruturaj-browserstack and @SavioBS629 on #441.

**A path value could retarget the request (High).** `encodeURIComponent` does not
escape a dot and `new URL()` resolves `.` and `..`, so a string path parameter
carrying `..` walked up a segment:

    update_test_case { project_id: "PR-1", test_case_id: ".." }
      -> PATCH /api/v2/projects/PR-1/        (the PROJECT, not a test case)

Reproduced before fixing. Worse than a failed call: the human had approved
"update this test case" and given a change_summary saying so, and the request
edits a different resource with that approval already banked. Eleven enabled
capabilities take a string path parameter with no pattern, enum or format, so
nothing stopped it — the old comment claiming type checking was the defence
held only for integer ids, and now says so.

`""`, `"."` and `".."` are refused; a dot INSIDE a value ("v1.2.3") still binds.

**Blank and fractional numbers were coerced (Low).** `Number("")` is 0, so a
blank id became id 0 and the call went wherever that resolved. `Math.trunc`
turned "1.9" into 1 silently. An id is either exact or it is the wrong id.

**Credentials could go out over plaintext (Medium).** The host comes from an
env var, account discovery or the index's own base_url, and nothing checked the
scheme before the access key was attached. Now refused unless https, or a local
host for development.

**Redaction had no cap, and two comments said otherwise (Medium, privacy).**
MAX_LENGTH is back at 512 and applied AFTER the rules, so a match cannot be
split across the boundary. redact.ts claimed a cap it had lost; MCPEventExtras
claimed "nothing here is free text written by a user" while `search_query` and
`change_summary` are exactly that. Both corrected to say what actually happens.

**Telemetry reason codes did not match the messages (Low).** `no_base_url`
matched "no base URL is configured" while config.ts throws "no host is
configured", so it never fired once. `capability_disabled` and
`unknown_endpoint` fell through to the generic bucket — the first is the
likeliest refusal on this build, with 46 of 244 capabilities withheld. Both
InvocationError paths now carry the capability identity, so a failure row can
be attributed. Every branch is pinned by a test against the message its source
actually produces.

**CodeQL alerts 4-7.** The first fix masked the account; CodeQL kept the alert
open, correctly — it follows the value from process.env through any transform,
and a mask is still a derived value on a path that did not need to carry it.
The account is now omitted from every stdout path, including `--show`, which
printed the pool file verbatim. The gitignored pool file still holds it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sarve-shreyas and others added 8 commits September 28, 2026 20:46
A generic five-tool MCP surface — listProducts, describeEntity,
searchCapability, describeCapability, invokeCapability — over a prebuilt
per-product index, replacing one hand-written tool per endpoint.

tm ships 244 capabilities, of which 198 are published. The other 46 are
withheld behind a `disabled` flag: everything that did not pass live
probing, plus the destructive tier. A disabled capability is filtered out
when the registry loads, so search, describe and the entity listings
cannot surface one and no reader downstream has to remember to check.

Capability names carry no version suffixes. Where dropping `_v1`/`_v2`
collided, the variants differ by ID FORM rather than API version, so the
prefixed-identifier variant keeps the plain name and the integer one takes
`_by_integer_id`. tests/live/CAPABILITY-RENAME-MAP.csv maps all 244
old names to new.

The index was verified against the live API rather than against the spec
it came from: 244 capabilities probed on preprod and two production
accounts, with the run records, payloads and findings under tests/live/.

Squashed onto main as a single commit. The development history is not
carried over — it contained real user email addresses in probe fixtures
and captured responses, and the working tree is clean of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(registry): record what was invoked, how the product answered, and why a call was refused

Registry telemetry recorded only a tool name at entry, and every error path returned
through `failed()` without recording anything.

- `trackMCP` takes an optional `extras` object (scalars only), merged into the row.
  Existing callers are unaffected.
- invokeCapability and describeCapability record the capability name, method, path, mode
  and product once resolved; invokeCapability adds `upstream_ok` and `upstream_status`
  when the product answers. `invoke()` returns 4xx/5xx rather than throwing, so `success`
  keeps its tool-level meaning and those two fields carry whether the call worked.
- searchCapability records result count, total matched, truncated, weak_match and
  coverage, plus the search text.
- A `refuse()` helper records and returns in one step, so every refusal carries a reason
  (unknown_capability, missing_parameter, bad_parameter_type, destructive_blocked,
  permission_not_granted, change_summary_missing, product_ambiguous, no_base_url).
  Refusals that never reach the network now produce a row.
- Free text (`search_query`, `change_summary`) passes through `redact()` first: emails,
  URL credentials, JWTs, opaque tokens, IPs, phone numbers and long digit runs are
  replaced by shape, with no word lists, then capped at 200 characters. It fails closed,
  dropping the field rather than sending raw text. Names have no shape and are not
  covered; the module documents this.

Registry rows stay identifiable by their tool names, and no field changes meaning for
any existing tool.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(registry): one telemetry row per invoke, not two

invokeCapability recorded once after resolving the capability and again once the product
answered, so a completed call produced two rows and counting invokes double-counted.

The first record existed to keep the capability name if the call never returned, but
every path between the two points already writes its own row through `refuse()`, so it
was redundant on every path. Removing it leaves one row per invoke, carrying both the
identity and the upstream result.

A call killed mid-flight now produces no row rather than a partial one, which is the
trade for a count that needs no filter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(registry): drop the length cap on recorded intent text

Redaction still runs on the full string, so truncation was removing detail without
adding protection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(registry): use the capability names the tm index publishes

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot download

Three corrections from live probing, all of them things the contract stated
confidently and wrongly.

`create_filter.filters` said only that its inner shape "is the same filter
object the entity search takes". It is not. The UI applies a saved view only
when its conditions are in the product's own filter-state shape — nested
`systemFields` / `extraSystemFields` / `matchAllMatchAny`, where each selected
option carries the project's numeric option id as `value` and its display name
as `label`. Nothing validates it, so the wrong shape saves and returns success,
then silently never applies. `extraSystemFields.issue_type` is required even
for a view that does not filter on linked issues, for the same reason: omit it
and the view saves but does nothing.

`list_filters.entity_type` was published as "optional, despite being published
as required", on the belief that omitting it returned one mixed list. It does
not — omitting it is a 400. And its rows carry only `id`, `name` and
`is_project_level`; the contract promised owner, entity_type, filters,
project_id and timestamps that never arrive.

`download_report` is withheld. It answers with a job channel this surface
cannot resolve, so it can never deliver the file it is named for — the same
test the other 46 were held to. tm now publishes 197 of 244.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL flagged two clear-text logs of a process-environment value
(code-scanning alerts 4 and 5), and both are right. The startup line printed
USERNAME directly, and the closing pool dump printed it again via pool.account.

The username is not a credential by itself, but it is half of the
`Api-Token: <username>:<access_key>` pair this surface authenticates with, and
stdout here reaches CI logs, terminal scrollback and screen shares — read by
more people than the shell that set the variable. The access key was never
logged; that part was already right.

Both sites now print a masked form (`ing…Xf`), which still distinguishes two
accounts, which is all the caller needed it for. The pool FILE keeps the full
value: it is gitignored, and knowing which account seeded a fixture is exactly
what you want when a probe result looks wrong.

Checked the sibling scripts for the same shape — replay-live.mts reads the same
variables but never logs them, and nothing else in scripts/ or src/ interpolates
a credential into a log line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, telemetry reasons

Review findings from @ruturaj-browserstack and @SavioBS629 on #441.

**A path value could retarget the request (High).** `encodeURIComponent` does not
escape a dot and `new URL()` resolves `.` and `..`, so a string path parameter
carrying `..` walked up a segment:

    update_test_case { project_id: "PR-1", test_case_id: ".." }
      -> PATCH /api/v2/projects/PR-1/        (the PROJECT, not a test case)

Reproduced before fixing. Worse than a failed call: the human had approved
"update this test case" and given a change_summary saying so, and the request
edits a different resource with that approval already banked. Eleven enabled
capabilities take a string path parameter with no pattern, enum or format, so
nothing stopped it — the old comment claiming type checking was the defence
held only for integer ids, and now says so.

`""`, `"."` and `".."` are refused; a dot INSIDE a value ("v1.2.3") still binds.

**Blank and fractional numbers were coerced (Low).** `Number("")` is 0, so a
blank id became id 0 and the call went wherever that resolved. `Math.trunc`
turned "1.9" into 1 silently. An id is either exact or it is the wrong id.

**Credentials could go out over plaintext (Medium).** The host comes from an
env var, account discovery or the index's own base_url, and nothing checked the
scheme before the access key was attached. Now refused unless https, or a local
host for development.

**Redaction had no cap, and two comments said otherwise (Medium, privacy).**
MAX_LENGTH is back at 512 and applied AFTER the rules, so a match cannot be
split across the boundary. redact.ts claimed a cap it had lost; MCPEventExtras
claimed "nothing here is free text written by a user" while `search_query` and
`change_summary` are exactly that. Both corrected to say what actually happens.

**Telemetry reason codes did not match the messages (Low).** `no_base_url`
matched "no base URL is configured" while config.ts throws "no host is
configured", so it never fired once. `capability_disabled` and
`unknown_endpoint` fell through to the generic bucket — the first is the
likeliest refusal on this build, with 46 of 244 capabilities withheld. Both
InvocationError paths now carry the capability identity, so a failure row can
be attributed. Every branch is pinned by a test against the message its source
actually produces.

**CodeQL alerts 4-7.** The first fix masked the account; CodeQL kept the alert
open, correctly — it follows the value from process.env through any transform,
and a mask is still a derived value on a path that did not need to carry it.
The account is now omitted from every stdout path, including `--show`, which
printed the pool file verbatim. The gitignored pool file still holds it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edit_test_run replaces whatever it is not sent, and the only thing that
knows what to send back is GET .../test-runs/{id}/edit/selection — the
read the product UI makes before every "Add to Run" edit. It was missing
from the index, so an agent could not use the full edit safely.

Add get_test_run_edit_selection (read, test_run, not permission-gated)
from teststack test_runs#get_selection, with a TestRunEditSelectionResponse
schema, and point edit_test_run at it: call it first, resend selection,
shared_selection, trtc_configuration_mappings and every run field.

Checked live on an open run (TR-5448072) and a closed one (TR-5448258).
Two things the source did not show went into the guidance:
- a run with no configurations still returns one default
  trtc_configuration_mappings entry (configuration_id null, select_all true);
- the stored selection is not kept in step with the add-only and remove
  calls — TR-5448258 still listed a removed case and a folder stored as
  select-all (unique_test_case_count 10 vs 9 cases). Resending it as-is
  would re-add cases, so the guidance says to compare counts first.

Published count 197 -> 198 of 245.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
close_test_run's guidance said a run could be reopened by editing its
run_state back, and update_latest_test_result advised reopening a closed
run. Neither is true.

Tried on TR-5448258 after closing it:
- update_test_run (v2 PATCH, run_state in_progress) -> 404 "Test Run ID is
  invalid", because the builds-backed lookup does not resolve closed runs.
- edit_test_run (v1 full edit, every field resent) -> 200 success, run still
  closed, closed_at unchanged.

Backend: testhub's run update throws "Closed Testruns can not be modified"
for any closed run (use_cases/testRun.js). The rearch builds path
(teststack builds_public_update_testrun_th -> testhub updateBuildsByUuid)
has no such guard and returns success instead — reported separately.

close_test_run now says closing is final and points to cloning the run to
keep testing; run_state on update_test_run, edit_test_run and
replace_test_run notes that closed is one-way and how each path answers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…fixed list

`loads the stored layout` pinned listProducts to ["tm"], so it broke the moment a
second index landed beside it in the fixture directory — which is the case the
stored layout exists for. One product could never prove that one-file-per-product
discovery works.

The expectation now reads the directory, so the test asserts what discovery found
rather than what it was written next to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sarve-shreyas
sarve-shreyas force-pushed the feat/tm-capability-registry branch from e28a886 to 7d7ac08 Compare September 28, 2026 15:19
@gaurav-singh-9227
gaurav-singh-9227 merged commit 5f34e4b into main Sep 28, 2026
4 checks passed
@sarve-shreyas
sarve-shreyas deleted the feat/tm-capability-registry branch September 28, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants