You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Block Darwin submission gate: synthetic promotion, fitness mismatch, unvalidated evidence and budget refunds #384
Independent review of 763bd80234be11942cfde10c30c0e79dcc9f5180 on feat/openenv-arena.
Decision: HOLD integration with submission automation and measured-win claims. These are synthetic, local software reproductions, not model-performance results. No source edits, GPU calls, deployment or Arena submission were made. External implementation remains with ruvultra.
1. P1: production gate signs synthetic promotion
The unmodified checked-in end-to-end test runs evaluator --dry-run, then gate --candidate-budget 1, and asserts promote:true, verified:true, reasons:[] plus successful pinned verification.
Rerun using committed Darwin dist, with a clean tracked working tree. Test, gate.
Default-budget rejection is incidental statistical rejection, not provenance enforcement. Reject dry-run/mock/proxy-only evidence in the production gate. Signature validation itself works; signatures authenticate bytes, not execution.
2. P1: absent provenance and duplicate seeds pass
Local decide() reproductions promote with only {seedBase:800000,attempts:32} provenance, absent model/source/trajectory hashes, or four duplicate seeds per block. Gate, confirmation.
Require complete typed provenance, frozen cell identities, unique matched seed sets and retained trajectory hashes; recompute classifications and aggregates from validated evidence.
3. P1: implemented fitness differs from the requested fixed-cell criterion
Fitness scores eligible mixed four-episode subsets, not fixed four-attempt cells with 1–3 full successes. Consequences:
[1,1,1,1,1,.5,.5,truncated] yields primary 3/7, mixedFromReasoning true and regressed false.
[0,.5,0,.5] yields primary 1 despite zero full successes.
Full gate reproduction: baseline 36 solved episodes; candidate .5 every fourth episode, otherwise 1, final episode completion_truncated. Matched provenance, confirmation seedBase800000, selection seedBase700000/attempts4/evaluated1. Result: primary 0.6329513623631271, regressed false, promote true, reasons [], eValue 25.62890625 (8 informative of 9 blocks).
Freeze four-attempt cell identities before sampling. Any truncation/format/infra contamination must invalidate that cell while retaining the denominator. Count mixed success only when 0 < solved < 4. Reward variance can be a separate descriptive metric, not a silently substituted objective. Apply identical eligibility rules in selection and confirmation.
Classification also prioritizes reward1 over explicit failure, accepts missing failure, and admits failure:null with finish_reason:length. Contradictions must fail closed. The retained science d2 fixture seeds600201–600203 are the previously diagnosed omitted-quarantined-median ambiguity; exclude that source-bound evidence from reasoning claims pending clarified-source remeasurement.
4. P1: cache and fresh-output evidence are insufficiently validated
Cache, evaluator, runner validation.
A cached block with wrong family/difficulty, duplicate seed7, rewards [1,.5,1,1], zero tokens, and no metadata/plan/receipt produces: regressed:false, primary:1, costPerWin:0, cachedCells:1, calls:0.
Fresh validation also accepts available:true with receipt:null and wrong model/seeds.
Use one strict validator on fresh output and cache reads/writes, binding cell identity, exact attempts/seeds, runner/environment/model/settings, receipts, trajectory digests and accounting.
5. P1: executed calls can be refunded after malformed output
Runner, evaluator fallback, budget refund.
A synthetic runner increments an invocation marker, writes malformed JSON to --output, and exits0. Three actual invocations under maxTotalNewCells1 were reported as calls [0,0,0], charged budget0.
Carry execution counts through all post-spawn read/parse/cache-write failures. Unknown executed work must never receive a zero-call refund.
6. P1: no enforced preregistration or persistent comparison ledger
Same evidence, evaluated13: default budget refuses (need14 blocks/have8); candidateBudget1 promotes.
alpha0.9 permits one winning block.
A truly predeclared single holdout could justify budget1; the implementation does not establish that history. Persist registration before outcomes, binding candidate, source/model/settings, alpha/lambda/budget and confirmation allocation; reject reused confirmations and posthoc weakening. Verification should bind expected rule/configuration/comparison identities, not merely a signing key. Fingerprint all source dependencies, including helpers.
Additional P2 findings
Budget lease: after owner A fails, B reclaims, but waiting C does not recheck current ownership and proceeds with allowance0 while B runs. Reproduced max_new_cells_exceeded:1>0. Loop and recheck ownership after every wait.
Cells: RUNNER_KNOBS is empty, so implemented suite_count_delta/sample_count_delta cannot yet evolve.
Short Trackio completions do not establish trainer thinking mode. Advertised /models metadata does not attest the actual checkpoint/template. Preserve those limitations explicitly.
Preregistered durable comparisons reject posthoc policy changes and reused confirmations.
Send a new exact SHA and focused test receipts, then independently rerun review before any GPU finalist or scheduler integration.
Existing passing tests encode several of the problematic behaviors. Gate tests10/10 and fitness/cells25passed/1skipped are not evidence these boundaries hold. No TypeScript typecheck or real provider success path was exercised in this review.
Review receipt, 9 October 2026, 18:35 Toronto. Exact source b8c3d38 on feat/openenv-arena. Owner remains ruvultra. No source edits or live submission.
New P1: the 978ab06 receipt fix is bypassed by reconcile(). In flywheel/arena-api.mjs, reconcile() unconditionally changes an existing refused receipt to recorded when GET finds any matching submission ID. flywheel/finish.mjs postProven() rejects only the current state refused, so follow() subsequently adopts the local digest even when the returned image/dataset are foreign.
Minimal local reproduction uses the real createArenaClient.reconcile and follow functions, overriding only getSubmission with a fixture:
await client.reconcile(receipt, receiptPath), then await follow(context,pending,0).
Observed: before refused, after recorded, refusal code retained, incumbentInstalled true, local digest installed, no needs-human note. Zero network calls; no credentials read.
This demonstrates the receipt state transition and adoption bug, not a claim that production Arena emitted this illustrative error code. A definite refusal must remain permanently nonadoptable. Separate observed remote lifecycle from immutable POST disposition; match available image/dataset metadata and require explicit reconciliation on conflicts. post_attempted means intent was persisted before network, not proof of acceptance.
Focused tests: node --test integrations/openenv-arena-darwin/flywheel/test/follow-receipt.test.mjs integrations/openenv-arena-darwin/flywheel/test/bootstrap-incumbent.test.mjs integrations/openenv-arena-darwin/flywheel/test/arena-submit.test.mjs
Result: 19 pass, 0 fail, 1 skip (hardcoded executor path for Python parity). They cover the original existing-ID case but not this refusal reconciliation path.
Acceptance: add the refused receipt → foreign validated GET → reconcile → follow regression; incumbent must remain unchanged, refusal history retained and human reconciliation required. Also reject conflicting image/dataset on uncertain outcomes.
Original #384 gate/fitness/cache/budget sources are unchanged by the branch diff since763bd802, so those findings remain HOLD. This is an unsigned review note, not a signed gate decision or submission approval.
Activity
ruvnet commentedon Oct 9, 2026
Review receipt, 9 October 2026, 18:35 Toronto. Exact source b8c3d38 on feat/openenv-arena. Owner remains ruvultra. No source edits or live submission.
New P1: the 978ab06 receipt fix is bypassed by reconcile(). In flywheel/arena-api.mjs, reconcile() unconditionally changes an existing refused receipt to recorded when GET finds any matching submission ID. flywheel/finish.mjs postProven() rejects only the current state refused, so follow() subsequently adopts the local digest even when the returned image/dataset are foreign.
Minimal local reproduction uses the real createArenaClient.reconcile and follow functions, overriding only getSubmission with a fixture:
Observed: before refused, after recorded, refusal code retained, incumbentInstalled true, local digest installed, no needs-human note. Zero network calls; no credentials read.
This demonstrates the receipt state transition and adoption bug, not a claim that production Arena emitted this illustrative error code. A definite refusal must remain permanently nonadoptable. Separate observed remote lifecycle from immutable POST disposition; match available image/dataset metadata and require explicit reconciliation on conflicts. post_attempted means intent was persisted before network, not proof of acceptance.
Focused tests: node --test integrations/openenv-arena-darwin/flywheel/test/follow-receipt.test.mjs integrations/openenv-arena-darwin/flywheel/test/bootstrap-incumbent.test.mjs integrations/openenv-arena-darwin/flywheel/test/arena-submit.test.mjs
Result: 19 pass, 0 fail, 1 skip (hardcoded executor path for Python parity). They cover the original existing-ID case but not this refusal reconciliation path.
Acceptance: add the refused receipt → foreign validated GET → reconcile → follow regression; incumbent must remain unchanged, refusal history retained and human reconciliation required. Also reject conflicting image/dataset on uncertain outcomes.
Original #384 gate/fitness/cache/budget sources are unchanged by the branch diff since763bd802, so those findings remain HOLD. This is an unsigned review note, not a signed gate decision or submission approval.