fix(registry): re-land TRA with only the in-scope endpoints by sarve-shreyas · Pull Request #469 · browserstack/mcp-server · GitHub
Skip to content

fix(registry): re-land TRA with only the in-scope endpoints - #469

Merged
gaurav-singh-9227 merged 4 commits into
mainfrom
reland/tra-capability-registry
Oct 8, 2026
Merged

gaurav-singh-9227 merged 4 commits into
mainfrom
reland/tra-capability-registry

Conversation

@sarve-shreyas

@sarve-shreyas sarve-shreyas commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Re-lands #456 with the problem it was reverted for fixed, and then narrows the TRA surface further on the back of live probing and the engineering test sheet.

#456 merged on 7 Oct and was reverted the same day (#466) because the TRA index shipped a large surface the product never meant to expose through MCP. This branch is cut from current main, reverts that revert, and applies the corrections on top — so the routing and OAuth work returns unchanged, and only the TRA index differs from what was already reviewed.

TRA: 502 → 254 capabilities

as authored 502
− out of scope −234
− collapsed duplicate routes −12
− service-to-service only −2
ships 254

Entities 39 → 31. No destructive capabilities remain. Modes: read 222, write 32.

1. Scope (−234)

Filtered against the agreed list of 268 method+path pairs: 268 listed, 268 matched, nothing unmatched in either direction.

Matching on both halves matters. A path appears once per method in that list, and matching on path alone keeps every other verb the index documents for it — the first attempt did exactly that and kept 18 extra, all 12 DELETEs among them.

Removed whole areas: project settings and members, integrations, CI triggers, tokens, groups, the user and plan surfaces.

2. Duplicate routes (−12)

Twelve /ext/v1 capabilities and their /api/v1 twins declared the same response schema entry — not merely the same shape — so each pair was one capability behind two names. The external route survives and takes the clean name.

The shared entry is good evidence rather than an assumption, because the index authors modelled divergence where they found it: builds have their own PublicBuildDetailsResponse precisely because that route differs.

Confirmed against the engineering sheet, where all 12 _via_public_api rows are marked Deploying = Yes and their internal twins No.

3. Service-to-service only (−2)

create_project and save_test_rca are X-Service-API-Key routes. A customer username and access key can never call them, so shipping them offers a capability that always fails. create_project's own guidance already said so.

Naming: _via_public_api is gone

It was never a reliable signal — of the 35 capabilities on /ext/v1, only 24 carried it. It appeared where a name collision forced it, not where the external API was, so its absence meant nothing. All 24 resolved three ways:

  • 12 collapsed — the surviving route takes the plain name; no marker needed once there is one capability per operation.
  • 5 renamed for what they actually do: get_build_summary (8 fields vs get_build_details' 17), replace_view (requires the whole object where update_view patches), list_project_builds_by_project_id / _by_name (different filter sets), improve_test_rca (takes an instruction the retrigger route does not), list_build_history_with_defect_counts.
  • 7 lost the suffix — no other capability held the name.

The ai-failure swap, worth reading twice

The internal route was list_build_ai_failure_test_runs_flat here but list_build_ai_failure_cluster_test_runs in the engineering test sheet. Dropping the suffix handed the external route the name the sheet already used for the internal one — one name, two endpoints, across two documents both treated as authoritative.

The internal now takes the sheet's name. The external becomes list_ai_failure_theme_test_runs, which is what it is: it accepts a theme or workflow id and, unlike its twin, cannot take testRunIds.

Guidance follows the names

Ten guidance strings pointed at capabilities that no longer exist — an agent following them would have asked for a name the registry cannot resolve.

get_build_summary claimed "the full public build record" while returning eight fields, three of which come back null. It now says what it gives, and carries what live probing found and nothing had documented:

status here is the build's LIFECYCLE ('finished'), not its outcome ('failed') as get_build_details reports it — the two routes answer different questions with the same field name.

With the suffix gone from the names, guidance is the only place left telling an agent which of a pair to reach for, so the remaining pairs now state their difference explicitly.

Everything else is #456 unchanged

Product routing (ambiguous prompts: 16.9% → 82.5% ask, failures 61.0% → 12.3% over a 201-prompt eval), the stateless signed routing token, product_choice removal, the central-OAuth extraction, and the askBrowserStackAI → capability-functions handoff rewrite. See #456 for detail.

Testing

1231 passing. All three products load: loadtesting 9 entities, tm 23, tra 31.

Worth knowing

  • Breaking against 2.2.0-beta.1. That build has the _via_public_api names, and its list_build_ai_failure_cluster_test_runs points at the external route where this one points at the internal. Anything pinning those names changes behaviour silently.
  • Routing shifts when a product halves. Words that settled on tra by weight of vocabulary now clash or settle elsewhere. The 201-prompt eval predates the trim and should be re-run before those numbers are quoted for this build.
  • Two rows in the engineering sheet still read #N/A, and neither is a scoping gap: list_build_ai_failure_test_runs_by_ids is the same endpoint driven by testRunIds, and get_repo_setting_by_build_name is list_repo_settings, removed as out of scope.

🤖 Generated with Claude Code

sarve-shreyas and others added 3 commits October 8, 2026 12:32
The index was authored from the full 534-endpoint inventory and shipped 502
capabilities, a large part of which the product never meant to expose through
MCP: project settings, members, integrations, CI triggers, tokens, groups and
the user and plan surfaces. That is what #456 was reverted for.

Filtered against the agreed list of 268 method+path pairs. Matching on BOTH
halves matters: a path appears once per method in that list, and matching on
path alone keeps every other verb the index happens to document for it — which
is how the first attempt kept 18 extra, all 12 DELETEs among them.

  capabilities  502 -> 268      schemas    273 -> 197
  entities       39 ->  31      responses  295 -> 163
                                paging      55 ->  46

No destructive capabilities remain: the agreed list contains no DELETE, so TRA
is now GET 207, POST 49, PUT 11, PATCH 1 — read 228, write 40, destructive 0.

The shared maps are garbage-collected transitively rather than left to rot:
`$response` from the kept capabilities, `$schema` from those responses, then
schema-to-schema until it settles. Zero dangling references afterwards, every
capability's entity resolves, all 268 names unique. The eight entities that lost
every capability — ci_trigger, integration, project_setting, rca_conversation,
redaction_rule, repo_setting, report_share, user_team — go with them, since an
entity describeEntity can answer for but nothing can act on is a dead end.

Routing shifts when a product halves: words that settled on tra by weight of
vocabulary now clash or settle elsewhere. The 201-prompt eval predates this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`build_id` is what the loader hands out for cache-busting, so it has to move
when the content does — and it still read `tra-v1.3-p0-eval-fixes` after the
index went from 502 capabilities to 268, which would have served a consumer the
old shape from cache under an id that claims to describe it.

Version and build id both bumped, top level and inside the product, keeping
TRA's own label convention. tm and loadtesting use `<sha>_<timestamp>` because
a harness generates them; this index is authored, so the label stays readable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

… URL they sit on

`_via_public_api` was never a reliable signal. Of the 35 capabilities on
/ext/v1 only 24 carried it — it appeared where a name collision forced it, not
where the external API was, so a reader could not trust its absence to mean
anything. All 24 are gone, three ways.

TWELVE COLLAPSED. The /ext/v1 route and its /api/v1 twin declared the same
response schema — the same entry, not merely the same shape — so the pair was
one capability behind two names. The external survives and takes the clean name.
That the authors modelled divergence where they found it, giving builds their
own PublicBuildDetailsResponse, is what makes the shared entry good evidence
rather than an assumption.

FIVE RENAMED, because those routes really do differ and the name should say how:
get_build_summary returns 8 fields where get_build_details returns 17;
replace_view requires the whole object where update_view patches;
list_project_builds_by_project_id and _by_name take different filter sets;
improve_test_rca takes an instruction the retrigger route does not.

SEVEN LOST THE SUFFIX, no other capability holding the name.

The ai-failure pair is the exception worth reading twice. The internal route was
`list_build_ai_failure_test_runs_flat` here and
`list_build_ai_failure_cluster_test_runs` in the engineering test sheet, so
dropping the suffix handed the EXTERNAL route the name the sheet already used
for the INTERNAL one — one name, two endpoints, two documents both treated as
authoritative. The internal now takes the sheet's name; the external becomes
`list_ai_failure_theme_test_runs`, which is what it is: it accepts a theme or
workflow id and, unlike its twin, cannot take testRunIds.

TWO REMOVED. create_project and save_test_rca are X-Service-API-Key routes; a
customer username and access key can never call them, so shipping them to a
customer surface offers a capability that always fails.

Prose follows the names: ten guidance strings pointed at capabilities that no
longer exist. get_build_summary claimed "the full public build record" while
returning eight fields, three of which come back null — it now says what it
gives, and carries the distinction live probing found: `status` here is the
build's lifecycle, not its outcome as get_build_details reports it, and each
route holds what the other lacks. With the suffix gone from the names, guidance
is the only place left telling an agent which of a pair to reach for.

254 capabilities, 31 entities, no destructive routes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gaurav-singh-9227
gaurav-singh-9227 merged commit 9709dbb into main Oct 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants