feat(instrumentation): record why a tool call failed by SavioBS629 · Pull Request #455 · browserstack/mcp-server · GitHub
Skip to content

feat(instrumentation): record why a tool call failed - #455

Merged
gaurav-singh-9227 merged 10 commits into
browserstack:mainfrom
SavioBS629:fix/telemetry-error-detail
Oct 5, 2026
Merged

gaurav-singh-9227 merged 10 commits into
browserstack:mainfrom
SavioBS629:fix/telemetry-error-detail

Conversation

@SavioBS629

@SavioBS629 SavioBS629 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Bug Fixes
    • Error reporting now captures useful diagnostic details when a tool returns an error, while masking sensitive information such as email addresses, IP addresses, credentials, and tokens.
    • Event reporting can continue when browser credentials are unavailable or cannot be processed.
    • Successful tool calls remain free of error details, and errors that throw continue to be reported through the existing path.

Tools report failure two ways: a few throw, most return `isError: true`.
Only the throwing path reached telemetry, so a returned failure wrote a row
saying the call failed and nothing about why.

Over the last 14 days that left 83% of failures with no diagnostic at all,
including every Test Management failure (1,485 across 50+ groups).

Carry the text the tool already showed the user into `error_message`, and add
`error_kind` to bucket failures as auth / not_found / rate_limited / validation
/ timeout / network / server_error so they can be charted.

The message is shape-redacted and capped at 300 chars. `success`, `outcome`
and the thrown-error path are unchanged, and the row count per call is
unchanged. Applies to every registered tool, local and remote.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

SavioBS629 and others added 5 commits October 1, 2026 13:43
Two corrections from review.

error_kind does not survive contact with real data. Run over 30 days of
production error messages the classifier returns "unknown" for 46.7% of them,
and rate_limited — the bucket that motivated it — matches 4 rows in 30 days.
Same failure as error_class: a column that reads mostly "unknown" invites
wrong conclusions. The message itself carries the detail; grouping can be
done in the query once we see what real text looks like.

The 300-character cap contradicted the standing rule not to truncate what we
log. A long error is exactly the one worth reading in full. Removed; a test
now pins that a 1,400-character message is stored whole.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
getBrowserStackAuth threw outside any guard, and the throw was swallowed by
the caller's catch, so the whole event was dropped. A config of {} wrote no
row at all while an absent config wrote one.

The auth header is now best effort; the row is sent either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The {20,} and {32,} alternation was quadratic on a long alphanumeric run:
50KB of them took 2.2s and blocked the event loop, which on a shared remote
pod stalls every concurrent request. Reachable today, since the circular
JSON failures already produce multi-KB stack traces.

Every quantifier now has an upper bound and the input is capped at 2000
characters, which is well clear of the longest message seen in production
at 513. Same input now takes 0.94ms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scrub claimed to mirror the capability registry but did not, and the gap
was the shape this server handles: a 20-character mixed-case access key
satisfied neither branch of the old token rule and passed through whole.
That is exactly how a credential reaches analytics, since an upstream auth
failure can echo the key back in its message.

Uses the registry's rules instead: the opaque-token lookaheads that require
both a digit and a letter, plus the phone and long-number rules that were
missing. Bounds are kept for the quadratic blowup. Also drops error_type's
synthetic "ToolError", which was the same value on every returned failure
and said nothing outcome does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SavioBS629

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

main had moved 44 commits, including the capability telemetry that added
ctx.extras to the same event object. Both changes belong: error_message and
extras are now spread together.

With redact.ts on main, the local scrub is gone and instrumentation uses the
registry's redact — one source of truth, as review asked. That also inherits
its fail-closed behaviour and its redact-before-truncate ordering.

Fixes a ReDoS in that shared redactor, found by this branch's tests: the
phone rule backtracked on a long digit run, taking 1,020ms for 40KB and
blocking the event loop for every concurrent caller. Quantifiers are bounded
and the scan window is capped; the same input now takes 2ms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the scanner API answers success: false without an errors array, the
thrown message interpolated undefined, and the catch below re-threw it as
"Failed to start scan: Unable to start scan: undefined" -- the exact text
behind roughly half of startAccessibilityScan's recorded failures, with the
real reason discarded. Fall back to the response body, and treat an empty
errors array the same way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SavioBS629 and others added 2 commits October 5, 2026 14:15
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gaurav-singh-9227
gaurav-singh-9227 merged commit 0773725 into browserstack:main Oct 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants