Repository navigation
[Dream Cycle 2026-10-01] flywheel-promotion: verifyMultiGenerationEvidence's frozen thresholds were float-noise-nondeterministic at the boundary + evals-verticals,bench scan #367
Description
Tonight's Rotation
DATE=2026-10-01, DEEP=flywheel-promotion, SCAN=evals-verticals,bench, SLOT=1, COMMIT=9ce8b8dd89045c3b9a1f809ae58f3589029db4a4, BRANCH=dream/2026-10-01-flywheel-promotion.
Build: full build succeeded (no wasm/NAPI degradation tonight — npm ci && npm run build clean across all 4 phases). LLM_EVAL=available (OPENROUTER_API_KEY present), not needed for tonight's candidate (pure unit-test proof, no live model calls). gh CLI auth is broken in this session (GH_TOKEN invalid per gh auth status), but GitHub MCP tools work directly (get_me confirms ruvnet). A raw gh gist create was additionally blocked by this session's own auto-mode policy classifier ("Public Data-Sharing Upload") — matching the fallback convention every night has used since 2026-09-09 (#299), the gist is committed in-repo only: GIST=LOCAL, docs/dream-cycle/2026-10-01-gist.md.
Ledger Check
Reviewed docs/dream-cycle/LEDGER.md and verified PR fates directly via search_pull_requests/pull_request_read (merged_at, not the sometimes-stale merged boolean — per this repo's own documented precedent), not inferred from ledger sparseness:
- The 2026-09-05→09-26 review backlog (escalated repeatedly 09-15 through 09-21) was fully cleared on 2026-09-26 — all 15 dream/* PRs back to 08-13 merged same day, confirmed via
pull_request_read. - 4 dream/* PRs currently open, all draft, all
mergeable_state: clean, 1-4 days old, none stale: dream(darwin-evolution): #359 close weight-eft's NaN-bypassable params-b/max-tokens safety gates (evaluated) #360 (09-27, darwin-evolution), dream(security-adversarial): #361 close --max-cost NaN spend-cap bypass (evaluated) #362 (09-28, security-adversarial), dream(host-adapters): #363 close kernel TrajectoryStore.readAll() uncaught-throw (evaluated) #364 (09-29, host-adapters), dream(generator-genome): #365 close defineHarness() tool/mcpServer collision gap (evaluated) #366 (09-30, generator-genome). Healthy state, no review-bottleneck bias needed tonight. - Scored the previous gist (2026-09-30, [Dream Cycle 2026-09-30] generator-genome: defineHarness() silent tool/mcpServer collision gap closed + router,turn-credit scan #365/dream(generator-genome): #365 close defineHarness() tool/mcpServer collision gap (evaluated) #366,
defineHarness()tool/mcpServer collision fix) per STEP 1.2 rubric: ~9/10 (grade-A test-suite evidence, ≥4 competitor rows incl. OpenAI Agents SDK/Google ADK/DSPy citations, specific next steps, valid witness, genuinely novel/externally-corroborated finding; estimated just over 1500 words based on issue-body length, -1). - Zero/low-merge-rate bias: not needed, backlog cleared.
- Duplicate-direction check (STEP 1.1): the
verifyReplayBundle/gateReExecutes/withSequentialEvidencewiring family has been hit 5x on this exact DEEP surface (08-16, 08-26, 09-01, 09-06, 09-16) — deliberately avoided touching that family again tonight. Separately, a NaN-input-validation bug class has now hit 2x as fixed candidates (09-27 dream(darwin-evolution): #359 close weight-eft's NaN-bypassable params-b/max-tokens safety gates (evaluated) #360 weight-eft, 09-28 dream(security-adversarial): #361 close --max-cost NaN spend-cap bypass (evaluated) #362 darwin-mode bench scripts) plus 1x disclosed-not-fixed (09-30 scan, turn-credit prior) — tonight's candidate is a different bug class (IEEE-754 threshold-comparison noise, CWE-697, not missing input validation) so it doesn't extend that count, but a disclosed-Infinity-sentinel finding in the same file (see below) was deliberately NOT selected to avoid becoming a 3rd/4th instance.
Deep Dive Findings (flywheel-promotion)
packages/flywheel/src/multigeneration-proof.ts (verifyMultiGenerationEvidence, PR #358, merged 2026-09-27 — zero Dream Cycle review until tonight) classifies multi-generation agent-harness-improvement evidence against DEFAULT_MULTIGEN_POLICY's documented "frozen thresholds" (ADR-318: "child successor yield exceeds parent successor yield by at least 5 percentage points"). Five sites compared raw floating-point deltas — (wins/attempts) - (wins/attempts), an IEEE-754 double subtraction of two division results — against those thresholds using strict </>=, with no epsilon tolerance.
Concrete, verified repro: attempts=20, parentWins=2, childWins=3 → 3/20 - 2/20 === 0.04999999999999999 in Node v22.22.0, a few ULPs short of the minImproverYieldDelta = 0.05 threshold, even though the true mathematical lift (1/20) equals the threshold exactly. A node -e integer search over attempts ∈ [10,200] found 542 such boundary-flip (parentWins, childWins, attempts) triples. A child that earns exactly the policy-mandated minimum successor-yield lift — 1 extra success out of 20 attempts, a realistic small-N case, not a contrived one — could be spuriously rejected (gen1:improver_yield_not_improved) purely as a function of which integer win/attempt pair produced a mathematically identical delta. This directly undermines ADR-318's "frozen thresholds" claim with a real nondeterminism.
Independent research (Deep Researcher role) confirmed this as a known class (CWE-697, floating-point comparison pitfall) and additionally flagged 4 other findings in the same file that were considered as candidates but not selected tonight — see "Candidates Considered" below and the evidence file.
Hypothesis
Given a GenerationEvidence whose improver-yield delta (or primary delta, relative-cost-reduction, or control-primary-margin) is mathematically exactly at a DEFAULT_MULTIGEN_POLICY threshold, when verifyMultiGenerationEvidence evaluates it, then the verdict must be identical regardless of which integer (wins, attempts) pair produced the delta — subject to: (a) existing passing/failing cases unaffected, (b) the epsilon tolerance stays far smaller than any policy threshold so it cannot mask a real shortfall, (c) 0 regressions in the existing suite.
Evaluation Receipt
- Baseline (pre-fix,
git stashon the source file only, independently reproduced by tonight's critic via its own stash cycle): new test fails non-vacuously with exactly['gen1:improver_yield_not_improved']; the other 8 tests in the file are unaffected (no collateral breakage) — confirming the test discriminates rather than being tautological. - Candidate:
packages/flywheelsuite 143→144/144 (+1, 0 regressions).tsc --noEmitclean.npm run build --workspace=@metaharness/flywheelclean. - Blast radius: zero downstream importers of
multigeneration-proof.tsoutsidepackages/flywheel/(grep -rlnacross allpackages/*/src+packages/*/__tests__, independently re-confirmed by the critic across the whole repo). This module is 4 days old and not yet wired into any production call site (ADR-318: "the existing RuFlo RSI mission should consume this verifier only after its execution budget, isolation, and evaluator ownership gates are satisfied"), so the blast radius is fully contained to its own test file tonight. evaluated: accepted
Darwin Results
Not run — not applicable. A float-comparison correctness fix to a verification primitive, not a tunable parameter under the ADR-071 Darwin mutation allowlist (routing weight, topology, prompt/memory policy, tool selection, tier/context policy, genome parameter).
Flywheel Evidence
OBSERVATION → MEASUREMENT (542-triple integer search + non-vacuous before/after test proof) → INFERENCE → HYPOTHESIS (frozen before the fix) → DECISION (ACCEPT). Full durable record with candidate-scoring table in docs/dream-cycle/2026-10-01-evidence.md, committed on this branch.
Reward Hack Check
Clear. Diff is additive (new comparator helper + consistent call-site substitution); no existing DEFAULT_MULTIGEN_POLICY threshold value changed; no existing test assertion weakened or removed (test diff is pure addition); no gold/benchmark data touched; no live model spend. Independently confirmed by the critic, which also verified the fix touches all 5 sibling threshold-comparison sites in the function (not just the originally-reported 4) and leaves none analogous unfixed/undisclosed.
Security Review
Not directly injection/credential/auth-sensitive, but this module exists specifically to resist gaming of self-improvement-claim evidence (ADR-318's whole purpose), so a nondeterministic boundary in its own "frozen" thresholds is a real correctness gap in a security-adjacent trust boundary. The independent critic additionally flagged (and we then acted on) that the initial THRESHOLD_EPS = 1e-9 was ~4500x looser than the real float-noise floor needed to fix the bug, since Outcome fields are arbitrary finite numbers, not restricted to win/attempt ratios (validOutcome places no such restriction) — a self-submitted-evidence adversary could in principle craft a genuine (non-noise) shortfall just inside an overly generous epsilon. Tightened to 1e-12 (still ~4500x the real noise floor, ~1e4x smaller than the smallest policy threshold minControlPrimaryMargin = 0.01) in direct response; full suite re-verified green after the change.
Scan Findings (evals-verticals, bench)
Reviewed packages/bench/src/draco/cost-efficiency.ts (pure-arithmetic Pareto-dominance comparator over already-measured cost/quality pairs) and JSON.parse call sites across packages/evals-toolcall/src/normalizer.ts and packages/evals-extract/src/normalizer.ts (model-output parsing, the highest-risk surface for malformed/adversarial input in these packages). Both are well-guarded: every JSON.parse call is try/catch-wrapped with a safe fallback, and cost-efficiency.ts's comparisons are internally consistent with no analogous float-boundary risk (its dominates check is an intentionally strict "costs strictly less" claim, not a frozen policy threshold). No actionable finding tonight — reported honestly per STEP 3's "if nothing new turns up, say so" guidance rather than manufacturing one.
Candidates Considered (not selected, scored in evidence file)
- Confirmation-set digests never bound to content (ADR-318 already half-discloses: "does not prove semantic non-overlap") — real, but a genuine fix needs a content-commitment redesign, out of scope for tonight's patch; good
security-adversarialcandidate. requireAllControlsas a caller-overridable policy flag enabling control-arm cherry-picking under a non-default policy — real but doesn't affect the shippedDEFAULT_MULTIGEN_POLICY; disclosed.- Hardcoded reviewer
verdict: 'ACCEPT'payload, no reject representation (enables reviewer-shopping, analogous to financial-audit "opinion shopping") — real design gap, needs a new verdict type; bigger than tonight's scope. -Infinitysentinel inrelativeCostReductionleaking unflagged intoderived[]whenbaseline.costPerWin === 0— deliberately not selected to avoid a 3rd/4th consecutive night on the "non-finite value silently flows into evidence" bug shape (09-27 dream(darwin-evolution): #359 close weight-eft's NaN-bypassable params-b/max-tokens safety gates (evaluated) #360, 09-28 dream(security-adversarial): #361 close --max-cost NaN spend-cap bypass (evaluated) #362) per STEP 1.1.
Competitors Reviewed
LangGraph, AutoGen, CrewAI, OpenAI Agents SDK/Codex, Claude Agent SDK ecosystem, DSPy/GEPA, plus (via "look past the fixed list" per STEP 3) RRSI (Google Research, arXiv:2609.24972), Darwin Gödel Machine (Sakana AI/UBC, arxiv.org/abs/2505.22954), Red Queen Gödel Machine (arxiv.org/abs/2606.26294), AlphaEvolve (DeepMind), MLflow/W&B model registries. Full table with grades/sources in the gist.
Gist
GIST=LOCAL — docs/dream-cycle/2026-10-01-gist.md (committed on this branch; raw gist-creation blocked by this session's auto-mode policy classifier, see note above).
Witness
session_commit : 9ce8b8dd89045c3b9a1f809ae58f3589029db4a4
report_sha256 : bc2b69d5574a36b1f850878696fea3ba897f8893fd92bd6ff46b5747cff56ac6
witness : c4c5281161ec46f5c80a24ae1545bfd32fc30b9aa46abd5744970842886239bb
Recommendation
ACCEPT — small (1 source file + 1 test file, ~40/-10 lines), independently critiqued (fresh-context adversarial pass reproduced before/after itself, found and we fixed 2 real minor gaps — EPS magnitude and a no-op test variable — verdict ACCEPT-WITH-CAVEATS → caveats resolved), non-vacuously proven, zero regressions, zero known downstream blast radius. Draft PR to follow.

Activity