{{ message }}
Repository navigation
feat(simulate): report repeated-trial runs as pass^1 / pass^k with a 95% interval (TH-8659) - #3555
Open
azain-commits wants to merge 3 commits into
Conversation
JayaSurya-27
requested changes
Oct 26, 2026
JayaSurya-27
left a comment
JayaSurya-27
left a comment
Member
There was a problem hiding this comment.
The scenario-level resampling is the right idea, but the interval method falls apart at the boundary, and a few header and tile numbers can contradict each other. Details are inline.
azain-commits
added a commit
that referenced
this pull request
Oct 26, 2026
…ader endpoint (TH-8659) Review follow-ups on #3555: - Interval: replace the percentile bootstrap, which collapses to zero width when scenario rates agree and under-covers near 100%, with a Wilson interval on pass^1 whose effective n comes from the spread of per-scenario rates (identical rates fall back to n = scenarios). Coverage at k=3, true rate 0.99 is ~0.94-0.98 for 10/20/50 scenarios. - pass^k: send pass_hat_k_passed and pass_hat_k_excluded; the header always names pass^k's scenario count, so a smaller set is visible. - Counts mode: backend sends pooled trials_passed / trials_evaluated and the header labels them "Trials passed" rather than pass^1. - Drop the pass^1 range from the pooled "Calls passed" tile and dashboard note, where the headline could sit outside it. - Add GET v3/test-executions/<id>/reliability/ so the run header no longer loads the full analytics payload.
…95% interval (TH-8659) - Replace the cluster-robust Wilson range with a scenario bootstrap: whole scenarios are resampled with their trials nested, never single trials. Withheld below 10 conclusive scenarios; fixed seed keeps it stable. - Reliability block gains pass_hat_1 (mean per-scenario pass rate over conclusive trials), pass_hat_k (share of scenarios passing all k trials, only when every scenario ran the same k > 1), k, inconclusive, and pass_hat_k_scenarios. Rows carry call_ids for trial drill-down. - Run header shows pass^1, pass^k, the range and inconclusive count; counts only below 10 scenarios. Scenario rows read "2/3 trials passed" and open their trials in the calls table. Trials picker recommends k >= 3. - Regenerate OpenAPI and frontend contracts.
…ader endpoint (TH-8659) Review follow-ups on #3555: - Interval: replace the percentile bootstrap, which collapses to zero width when scenario rates agree and under-covers near 100%, with a Wilson interval on pass^1 whose effective n comes from the spread of per-scenario rates (identical rates fall back to n = scenarios). Coverage at k=3, true rate 0.99 is ~0.94-0.98 for 10/20/50 scenarios. - pass^k: send pass_hat_k_passed and pass_hat_k_excluded; the header always names pass^k's scenario count, so a smaller set is visible. - Counts mode: backend sends pooled trials_passed / trials_evaluated and the header labels them "Trials passed" rather than pass^1. - Drop the pass^1 range from the pooled "Calls passed" tile and dashboard note, where the headline could sit outside it. - Add GET v3/test-executions/<id>/reliability/ so the run header no longer loads the full analytics payload.
…r rebase on dev (TH-8659)
azain-commits
force-pushed
the
azain/th-8659-report-repeated-trial-runs-as-pass1-passk-with-a-95-interval
branch
from
October 26, 2026 13:02
286ce21 to
26245c5
Compare
JayaSurya-27
approved these changes
Oct 26, 2026
JayaSurya-27
left a comment
JayaSurya-27
left a comment
Member
There was a problem hiding this comment.
Thanks, all five points are fixed. The Wilson interval behaves well at the boundaries, and the header and tile numbers now agree.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes TH-8659.
What
Repeated-trial runs now report reliability and uncertainty instead of a bare pass rate.
Backend (
simulate/services/run_reliability_v3.py)scenario_bootstrap_intervalreplaces the cluster-robust Wilson range. It resamples whole scenarios with their trials nested, never single trials. 2000 resamples, fixed seed so the range is stable across loads. Returnsnullbelow 10 conclusive scenarios. Intended for reuse by compare (TH-8657).reliabilityonGET /simulate/v3/test-executions/{id}/analytics/gains:pass_hat_1: mean over scenarios of passed ÷ conclusive trialspass_hat_k: share of scenarios that passed all k trials;nullunless every scenario ran the same k and k ≥ 2k,inconclusive(error + inconclusive trials, excluded from every denominator),pass_hat_k_scenarioscall_idsfor drilling into trialspass_rate_intervalis now{low, high, scenarios, evaluated, resamples};effective_nandclustersare removed. All in-repo consumers updated. OpenAPI + frontend contracts regenerated.Frontend
Decisions worth a look
±N ptsmargin is replaced by the pass^1 range, since the tile's pooled rate can differ from pass^1.Testing
bin/test simulate/tests/test_run_reliability_v3.py simulate/tests/test_analytics_functional.py: pass (reliability tests rewritten before the implementation).test_dashboard_drop_off.pythat pass when the file runs alone, with and without this change.vitest runonruns/detailandscenarios: 435/435. ESLint + Prettier clean.