Block Darwin submission gate: synthetic promotion, fitness mismatch, unvalidated evidence and budget refunds · Issue #384 · ruvnet/metaharness · GitHub
Skip to content

Block Darwin submission gate: synthetic promotion, fitness mismatch, unvalidated evidence and budget refunds #384

Open

Description

@ruvnet

Independent review of 763bd80234be11942cfde10c30c0e79dcc9f5180 on feat/openenv-arena.

Decision: HOLD integration with submission automation and measured-win claims. These are synthetic, local software reproductions, not model-performance results. No source edits, GPU calls, deployment or Arena submission were made. External implementation remains with ruvultra.

1. P1: production gate signs synthetic promotion

The unmodified checked-in end-to-end test runs evaluator --dry-run, then gate --candidate-budget 1, and asserts promote:true, verified:true, reasons:[] plus successful pinned verification.

node --test integrations/openenv-arena-darwin/test/e2e.test.mjs
# tests 1; pass 1; fail 0

Rerun using committed Darwin dist, with a clean tracked working tree. Test, gate.
Default-budget rejection is incidental statistical rejection, not provenance enforcement. Reject dry-run/mock/proxy-only evidence in the production gate. Signature validation itself works; signatures authenticate bytes, not execution.

2. P1: absent provenance and duplicate seeds pass

Local decide() reproductions promote with only {seedBase:800000,attempts:32} provenance, absent model/source/trajectory hashes, or four duplicate seeds per block. Gate, confirmation.
Require complete typed provenance, frozen cell identities, unique matched seed sets and retained trajectory hashes; recompute classifications and aggregates from validated evidence.

3. P1: implemented fitness differs from the requested fixed-cell criterion

Fitness scores eligible mixed four-episode subsets, not fixed four-attempt cells with 1–3 full successes. Consequences:

  • [1,1,1,1,1,.5,.5,truncated] yields primary 3/7, mixedFromReasoning true and regressed false.
  • [0,.5,0,.5] yields primary 1 despite zero full successes.
  • Full gate reproduction: baseline 36 solved episodes; candidate .5 every fourth episode, otherwise 1, final episode completion_truncated. Matched provenance, confirmation seedBase800000, selection seedBase700000/attempts4/evaluated1. Result: primary 0.6329513623631271, regressed false, promote true, reasons [], eValue 25.62890625 (8 informative of 9 blocks).

Freeze four-attempt cell identities before sampling. Any truncation/format/infra contamination must invalidate that cell while retaining the denominator. Count mixed success only when 0 < solved < 4. Reward variance can be a separate descriptive metric, not a silently substituted objective. Apply identical eligibility rules in selection and confirmation.

Classification also prioritizes reward1 over explicit failure, accepts missing failure, and admits failure:null with finish_reason:length. Contradictions must fail closed. The retained science d2 fixture seeds600201–600203 are the previously diagnosed omitted-quarantined-median ambiguity; exclude that source-bound evidence from reasoning claims pending clarified-source remeasurement.

4. P1: cache and fresh-output evidence are insufficiently validated

Cache, evaluator, runner validation.
A cached block with wrong family/difficulty, duplicate seed7, rewards [1,.5,1,1], zero tokens, and no metadata/plan/receipt produces:
regressed:false, primary:1, costPerWin:0, cachedCells:1, calls:0.
Fresh validation also accepts available:true with receipt:null and wrong model/seeds.
Use one strict validator on fresh output and cache reads/writes, binding cell identity, exact attempts/seeds, runner/environment/model/settings, receipts, trajectory digests and accounting.

5. P1: executed calls can be refunded after malformed output

Runner, evaluator fallback, budget refund.
A synthetic runner increments an invocation marker, writes malformed JSON to --output, and exits0. Three actual invocations under maxTotalNewCells1 were reported as calls [0,0,0], charged budget0.
Carry execution counts through all post-spawn read/parse/cache-write failures. Unknown executed work must never receive a zero-call refund.

6. P1: no enforced preregistration or persistent comparison ledger

Gate configuration can be weakened after outcomes:

  • Same evidence, evaluated13: default budget refuses (need14 blocks/have8); candidateBudget1 promotes.
  • alpha0.9 permits one winning block.

A truly predeclared single holdout could justify budget1; the implementation does not establish that history. Persist registration before outcomes, binding candidate, source/model/settings, alpha/lambda/budget and confirmation allocation; reject reused confirmations and posthoc weakening. Verification should bind expected rule/configuration/comparison identities, not merely a signing key. Fingerprint all source dependencies, including helpers.

Additional P2 findings

  • Budget lease: after owner A fails, B reclaims, but waiting C does not recheck current ownership and proceeds with allowance0 while B runs. Reproduced max_new_cells_exceeded:1>0. Loop and recheck ownership after every wait.
  • Cells: RUNNER_KNOBS is empty, so implemented suite_count_delta/sample_count_delta cannot yet evolve.
  • Short Trackio completions do not establish trainer thinking mode. Advertised /models metadata does not attest the actual checkpoint/template. Preserve those limitations explicitly.

Acceptance before another readiness review

  1. Negative tests reject synthetic, missing/contradictory provenance, duplicate seeds, contaminated cells, all-unsuccessful variance, and malformed cached/fresh trajectories.
  2. Executed malformed-output calls exhaust the configured call budget; concurrent waiters cannot bypass ownership.
  3. Preregistered durable comparisons reject posthoc policy changes and reused confirmations.
  4. Send a new exact SHA and focused test receipts, then independently rerun review before any GPU finalist or scheduler integration.

Existing passing tests encode several of the problematic behaviors. Gate tests10/10 and fitness/cells25passed/1skipped are not evidence these boundaries hold. No TypeScript typecheck or real provider success path was exercised in this review.

Activity

  1. ruvnet commented on Oct 9, 2026

    @ruvnet
    OwnerAuthor

    Review receipt, 9 October 2026, 18:35 Toronto. Exact source b8c3d38 on feat/openenv-arena. Owner remains ruvultra. No source edits or live submission.

    New P1: the 978ab06 receipt fix is bypassed by reconcile(). In flywheel/arena-api.mjs, reconcile() unconditionally changes an existing refused receipt to recorded when GET finds any matching submission ID. flywheel/finish.mjs postProven() rejects only the current state refused, so follow() subsequently adopts the local digest even when the returned image/dataset are foreign.

    Minimal local reproduction uses the real createArenaClient.reconcile and follow functions, overriding only getSubmission with a fixture:

    1. Create pending {submissionId:"local-collision", requestSha256:"a".repeat(64), receiptPath, genome:baselineGenome(), date:"2026-10-09"}.
    2. Write receipt {submission_id:"local-collision",request_sha256:"a".repeat(64),post_attempted:true,state:"refused",http_status:409,error_code:"SUBMISSION_ALREADY_EXISTS",slot_consumed:false}.
    3. client.getSubmission = async () => ({submission_id:"local-collision",state:"validated",images:[{submitted:"ghcr.io/other/image:v1",resolved:"ghcr.io/other/image@sha256:"+"b".repeat(64)}],dataset:{repo:"other/data",revision:"c".repeat(40)}}).
    4. await client.reconcile(receipt, receiptPath), then await follow(context,pending,0).
      Observed: before refused, after recorded, refusal code retained, incumbentInstalled true, local digest installed, no needs-human note. Zero network calls; no credentials read.

    This demonstrates the receipt state transition and adoption bug, not a claim that production Arena emitted this illustrative error code. A definite refusal must remain permanently nonadoptable. Separate observed remote lifecycle from immutable POST disposition; match available image/dataset metadata and require explicit reconciliation on conflicts. post_attempted means intent was persisted before network, not proof of acceptance.

    Focused tests: node --test integrations/openenv-arena-darwin/flywheel/test/follow-receipt.test.mjs integrations/openenv-arena-darwin/flywheel/test/bootstrap-incumbent.test.mjs integrations/openenv-arena-darwin/flywheel/test/arena-submit.test.mjs
    Result: 19 pass, 0 fail, 1 skip (hardcoded executor path for Python parity). They cover the original existing-ID case but not this refusal reconciliation path.

    Acceptance: add the refused receipt → foreign validated GET → reconcile → follow regression; incumbent must remain unchanged, refusal history retained and human reconciliation required. Also reject conflicting image/dataset on uncertain outcomes.

    Original #384 gate/fitness/cache/budget sources are unchanged by the branch diff since763bd802, so those findings remain HOLD. This is an unsigned review note, not a signed gate decision or submission approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions