{{ message }}
Add Live Debugger runtime benchmark coverage#429
Open
watson wants to merge 1 commit into
Open
Conversation
Contributor
Author
Live Debugger Runtime BenchmarkSDK-loaded dormant-probe runtime overhead, measured against an uninstrumented bundle in the same browser session.
Full diagnosticsRaw samples are in the |
watson
force-pushed
the
watson/DEBUG-5787/add-bench
branch
5 times, most recently
from
3a9f3d6 to
1296572
Compare
This was referenced Jun 24, 2026
watson
force-pushed
the
watson/DEBUG-5787/add-bench
branch
2 times, most recently
from
d80cc4f to
ba9d257
Compare
This was referenced Jun 29, 2026
The browser Live Debugger instrumentation needs repeatable runtime overhead checks before transform changes land. Add an opt-in Playwright benchmark that compares baseline, control, and instrumented workloads in the same browser session with dormant probe hooks installed. Report conservative per-call overhead bounds with confidence intervals, A/A diagnostics, block bootstrap checks, and PR comment output. Wire the benchmark into CI as a non-blocking job, share Playwright setup between jobs, upload raw samples as artifacts, and document how contributors should run and interpret the benchmark.
watson
force-pushed
the
watson/DEBUG-5787/add-bench
branch
from
ba9d257 to
0168305
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why?
This adds repeatable browser runtime benchmark coverage for dormant Live Debugger instrumentation. The goal is to give Live Debugger browser changes a controlled way to track whether instrumentation changes introduce measurable overhead in real browsers.
How?
Adds a Playwright benchmark under
packages/tests/src/bench/liveDebuggerRuntimewith Tiny and Hot workload shapes, baseline/control/instrumented variants, and SDK-like dormant probe hooks installed in the page.Adds a custom reporter that computes conservative per-call overhead bounds, 95% confidence intervals, A/A diagnostics, moving-block bootstrap intervals, autocorrelation diagnostics, browser failures, raw JSON output, and a PR comment body.
Adds
yarn workspace @dd/tests bench:live-debugger:runtime, a dedicated Playwright config, result artifacts, a non-blocking CI job, a shared Playwright setup action, and Live Debugger contributor docs explaining how to run and interpret the benchmark.For more details, see the "Runtime benchmark" section in the added
CONTRIBUTING.mdfile in this PR.Validation
The benchmark runs in CI as a non-blocking job and uploads raw samples plus the generated report artifacts for review.
Benchmark note
The current benchmark numbers show a significant dormant-instrumentation overhead in Safari. Most of this is fixed in stacked PR #438.