ruvnet · Oct 21, 2026 · Oct 16, 2026 · Oct 16, 2026 · Oct 16, 2026
diff --git a/README.md b/README.md
@@ -61,6 +61,8 @@ Output is an npm-publishable `.zip` with **your name on it, your branding, your
 
 ### New
 
+* Bounded contrastive harness development: `@metaharness/darwin/contrastive` adds scoped proposals, actual probes, selective composition and metered experiment memory behind immutable caller policy. [API guide](packages/darwin-mode/CONTRASTIVE.md). Independent paired confirmation adds unseen-task, statistical and regression checks. Public intent-classification evidence remains separate from general agent performance claims.
+
 [Watch Darwin evolve a harness](docs/visual-walkthrough.md#evolve) · [Explore model routing](docs/visual-walkthrough.md#routing) · [Review experimental extensions](docs/visual-walkthrough.md#extensions)
 
 - **Run an experimental ARC-AGI-3 harness from the ChatGPT UI.**

diff --git a/docs/adrs/ADR-319-bounded-contrastive-development.md b/docs/adrs/ADR-319-bounded-contrastive-development.md
@@ -0,0 +1,52 @@
+# ADR-319: Bounded contrastive harness development
+
+Status: accepted for an opt-in experimental API. Date: 2026-10-16.
+
+## Outcome and scope
+
+Add `@metaharness/darwin/contrastive`: two evidence-scoped editors, actual candidate probes, selective composition, bounded experiment memory and a caller-owned editor bridge. Existing Darwin CLI/evolve behavior is unchanged. The business goal is fewer repeated failures without paying for exploration on every task.
+
+Inputs are a declarative harness, task, environment, immutable capability envelope, scoped library, explicit budgets and trusted execution adapters. Outputs are a selected development candidate, execution fingerprints, receipts and costs. No module may publish, promote, execute embedded code, obtain credentials or start agents itself.
+
+The design is an original MIT implementation informed by [Harness²](https://arxiv.org/abs/2610.14224) and its [reference release](https://github.com/google-research/harness2). No upstream source was copied. Its similarly named Meta-Harness baseline is a separate project.
+
+## Control flow
+
+1. Validate and freeze all serializable inputs. Reject a surface-independent evaluator.
+2. Reserve the complete per-call ceiling before a baseline execution.
+3. Produce separate invariant and domain edits grounded in component-matching observation IDs. No development score is passed to editors.
+4. Execute the candidate under the same frozen task and environment. Match the adapter's applied digest to the candidate digest. Reuse identical configurations within this round.
+5. If both scopes improved, ask the composer to select only already supported edits. Execute that combination; retain the better individual if composition regresses.
+6. On malformed data, timeout, exceeded admission budget or adapter failure, return the baseline. No automatic retries or production state changes.
+7. Optionally retain the improved executed document as an experimental memory record. Retrieval filters tenant, project and environment before lexical ranking. Later tasks start from their caller's baseline.
+
+## Architecture and trust
+
+`engine.ts` owns admission and selection. `validation.ts` owns JSON bounds, schema, component dependencies, evidence and fingerprints. `editors.ts` bridges a real model client or Ruflo executor callback. Ruflo task records alone are not execution. `memory.ts` creates portable immutable records; persistence and RuVector indexing are deployment responsibilities.
+
+Sequential admission is deliberate: this first release favors unambiguous cancellation and spend accounting over concurrent provider work. Digest reuse avoids redundant probes. Monetary ceilings use integer nanodollars. Unknown-cost failures retain the reservation; known over-ceiling charges are recorded honestly and invalidate the round.
+
+A digest echo is trusted adapter attestation, not cryptographic proof of execution. Callbacks run in the same process and must enforce actual provider limits, cancellation, tenant authentication and any OS isolation. Timeout stops acceptance of late results; it cannot forcibly stop a dishonest callback. JSON validation does not certify script content. Any adapter executing generated code must apply existing Darwin safety gates and an appropriate OS sandbox.
+
+The library is data, not authority. Entries cannot expand capabilities or authorize promotion. Retrieval rejects unknown schema fields and invalid promotion/status values. Free-form content remains untrusted. Protect stored records with deployment access controls; a content digest does not authenticate its author.
+
+## Acceptance and evidence
+
+| Requirement | Evidence command or case |
+| --- | --- |
+| Candidate affects execution | Parser adapter reads the supplied components; wrong digest rejected |
+| Immutable scope and safety | Frozen callback inputs; cross-scope and capability edits rejected |
+| Admission before effects | Seven-call success and six-call fallback; exact decimal boundary test |
+| No late-result adoption | Timeout and late promise tests |
+| Evidence constrained composition | Invalid citations, unsupported edits and regressing composition tests |
+| Memory isolation | Tenant/project/environment checks and extra grading-field rejection |
+| Usable provider integration | Live provider editors with a deterministic synthetic executor |
+| Reproducibility | Source/build manifest, 100 unseen generated inputs and 32 control seeds |
+
+Run `npm run build -w @metaharness/darwin` then `npm test -w @metaharness/darwin -- --maxWorkers=2 --minWorkers=1`. Run `node packages/darwin-mode/bench/contrastive/run.mjs --out=/tmp/contrastive.json` for the synthetic mechanism benchmark. Live editor smoke is separately opt-in: `node packages/darwin-mode/bench/contrastive/live-pilot.mjs --live --out=/tmp/contrastive-live.json` with an authorized `OPENROUTER_API_KEY` in the environment.
+
+The baseline and sampling controls use the same synthetic task generator. Equal logical operations are not equal real inference dollars. These results do not establish improvement over current Darwin on real agent tasks, general transfer, or the proposed five-point real-task gate. A production performance claim still requires an independent, preregistered real-task comparison with total measured inference costs.
+
+## Release and rollback
+
+Release this additive API as `@metaharness/darwin@0.11.0`, with the existing default path preserved. Publish only the reviewed tarball after type checks, package regression tests, adversarial checks, production dependency audit and fresh-install smoke. Bind the release evidence to its source commit and npm integrity. Rollback is version pinning to `0.10.3`; no storage migration or production promotion occurs automatically.
diff --git a/docs/adrs/ADR-320-independent-contrastive-confirmation.md b/docs/adrs/ADR-320-independent-contrastive-confirmation.md
@@ -0,0 +1,48 @@
+# ADR 320: Independent paired confirmation
+
+Status: accepted for the optional Darwin contrastive API
+Date: 2026-10-16
+
+## Decision and value
+
+Development scores select hypotheses. They cannot establish that the selected configuration improves unseen tasks. Version 0.12.0 adds a separate confirmContrastive API that evaluates one frozen candidate against its baseline on a declared confirmation set.
+
+The gate provides an explicit experimental decision before a caller considers retaining or deploying a change. It never authorizes production promotion.
+
+## Inputs and invariants
+
+1. The caller supplies frozen baseline and candidate documents, tenant and project scope, an environment definition, a capability envelope, development task IDs and confirmation task IDs.
+2. Task IDs are unique within each partition and disjoint between partitions. Deployments should derive IDs from normalized task content; changing an ID does not make reused content unseen.
+3. A trusted evaluator executes both configurations for each case under the same model, tool versions, task inputs, output limits and provider conditions.
+4. Every successful pair binds to task identity, both applied document digests, tenant, project and environment.
+5. Call ceilings and spend reservations are admitted before an effect. timeoutMs is one deadline for the entire confirmation phase, not a fresh timeout per pair. Failed responses cannot authorize confirmation. Cancellation rejects late results; the adapter remains responsible for stopping actual effects.
+6. All declared cases must finish. There is no early acceptance and no editor callback during confirmation.
+7. The result always carries promotion: not-authorized.
+
+The evaluator retains answers and task payloads in its closure. The core receives passed booleans and execution metadata. This is a trust contract, not isolation from malicious callbacks.
+
+## Decision rule
+
+Count wins, losses and ties using paired binary outcomes. Apply an exact one-sided sign test to discordant pairs. Confirm only when the full set passes the caller's minimum sample, minimum wins, minimum accuracy gain, maximum regressions and significance thresholds.
+
+The recommended initial policy is at least 20 cases, at least 5 wins, zero regressions, at least 0.05 absolute accuracy gain and alpha 0.05. Five wins and zero losses produce p = 0.03125; a tie cannot pass. This policy is intentionally conservative.
+
+The statistical interpretation assumes independent task pairs and a prespecified candidate and policy. The API cannot police repeated use across processes. A deployment must maintain a durable confirmation-set registry, spend alpha across repeated hypotheses, and retire a set after inspecting its results. Do not tune prompts against confirmation failures and reuse the same set as new evidence.
+
+## Alternatives
+
+Reusing the development score costs less but measures selection on the same observations. A weighted aggregate alone can conceal regressions. Automatic promotion would combine experimental evidence with release authority. Those options are excluded.
+
+Confirmation is separate from runContrastive so existing callers retain behavior and deployments can reserve different budgets and privileges for development and independent evaluation. A declared maximum cost is not a claim that two methods spent exactly equal dollars.
+
+## Validation
+
+Unit and adversarial tests cover successful paired confirmation, ties, forbidden regressions, insufficient evidence, overlapping task IDs, duplicate task IDs, scope and digest mismatch, invalid values, cancellation and reservation limits.
+
+The public task experiment uses pinned CLINC150 human utterances and frozen development and test partitions. It reports provider costs and latency and compares a competent frozen baseline, the selected contrastive configuration and an independent prompt search control. The comparison plan requires contrastive versus baseline and contrastive versus independently selected control, with alpha 0.025 per comparison. Both must pass to support an improvement claim over both alternatives. Fresh confirmation predictions may be shared across the two comparisons, with costs counted once. No development execution cache is reused.
+
+This is a small public-data pilot. It does not prove private transfer, resistance to training contamination or general coding performance.
+
+## Rollback and remaining authority
+
+Pin @metaharness/darwin to 0.11.0 to omit the new API. Existing evolve and runContrastive defaults do not change. The deployment owner controls secrets, execution isolation, heldout data custody, repeated-testing policy and production releases.
diff --git a/docs/evidence/confirmation-0.12.0/README.md b/docs/evidence/confirmation-0.12.0/README.md
@@ -0,0 +1,37 @@
+# Independent confirmation release evidence
+
+Version: @metaharness/darwin 0.12.0. Date: 2026-10-16 UTC.
+
+The new confirmation API tests a frozen candidate against a frozen baseline on disjoint task IDs under a frozen policy. It returns an evidence verdict and never authorizes promotion. Scope and execution digests must match. All declared pairs must complete within one total deadline and conservative monetary reservations. The one-sided exact paired test is evaluated only after the declared set completes.
+
+## Observed public benchmark
+
+One prespecified CLINC150 pilot used 24 development utterances and 40 separate confirmation utterances across eight finance intents. Dataset provenance, model, provider route, budgets, source hashes and comparison policies were frozen before the first model call. Candidate documents were frozen before confirmation. There was one live run and no tuning or rerun after confirmation.
+
+| Measure | Baseline | Contrastive candidate | Control |
+| --- | ---: | ---: | ---: |
+| Development correct | 23/24 | 24/24 | 23/24 |
+| Confirmation correct | 38/40 | 38/40 | 38/40 |
+| Cost of 40 confirmation predictions | $0.001085832 | $0.001256124 | $0.001085832 |
+
+Both superiority comparisons had zero paired wins, zero losses and p = 1. Both rejected with insufficient-wins. The control selected the unchanged baseline, so its confirmation predictions were shared and billed once. The candidate cost 15.68% more for the same observed accuracy. Keep the baseline. This experiment demonstrates a failed transfer of development gain; it does not demonstrate general accuracy improvement.
+
+Total provider usage reported $0.007642223 for 253 requests. Conservative client reservations totaled $0.596 against a $1 ceiling, with zero unresolved attempts. Requested route: google-vertex/eu, model google/gemini-2.5-flash-lite. Actual usage charges and reserved ceilings are different quantities.
+
+This is a small public subset, potentially seen during pretraining. It is not the full CLINC150 benchmark or a general agent evaluation. Statistical assumptions include independent pairs and prespecified candidates and policies. Repeated experiments require additional error control.
+
+## Verification
+
+830 tests passed, 19 existing skips. This includes 99 focused confirmation tests. Clean installation, TypeScript compilation, deterministic adapter checks and production dependency audit passed. The production npm audit reported zero vulnerabilities at audit time. The independent security review is in packages/darwin-mode/bench/confirmation/security-review.md.
+
+Source/build hash for the live experiment: cc99a5833a39f2f6d9533faac08f249cbfd8cef54e31bc6e8dea97082d7b19eb. Local run source commit: 7b7cb54bc5afae0ee9a6a690b632ccc8662ce5ab. The freeze asset lists individual source and distribution file hashes, allowing verification against the published source tree.
+
+The GitHub release attaches full freeze, candidate and result JSON files. live-benchmark-summary.json records their hashes and costs. validation.json records verification scope. The final package acceptance is also attached to the release.
+
+## Consumer acceptance test
+
+In a fresh temporary project, install @metaharness/darwin@0.12.0 and copy acceptance.mjs into that project, then run node acceptance.mjs. It executes 20 paired cases through the installed API, confirms six wins with p = 0.015625, rejects a mismatched execution envelope and verifies that promotion remains unauthorized. This is an API acceptance test with a deterministic evaluator, not a model performance claim.
+
+## Repository CI
+
+The package release is independently validated. PR #407 remains unmerged because existing Rust dependency resolution fails on Candle 0.8 and yanked zip 1.x. Issue #408 tracks that blocker. This release does not claim Rust workspace CI or cargo audit success.
diff --git a/docs/evidence/confirmation-0.12.0/acceptance.mjs b/docs/evidence/confirmation-0.12.0/acceptance.mjs
@@ -0,0 +1,31 @@
+import assert from 'node:assert/strict';
+import { confirmContrastive, pairedSignPValue } from '@metaharness/darwin/contrastive';
+const baseline={components:[{id:'rules',scope:'invariant',content:{corrects:false}}]};
+const candidate={components:[{id:'rules',scope:'invariant',content:{corrects:true}}]};
+let calls=0;
+const evaluatePair=async input=>{
+ calls++;
+ const i=Number(input.taskId.split(':')[1]);
+ const passes=h=>i>=6||h.components[0].content.corrects;
+ return {tenantId:input.tenantId,projectId:input.projectId,
+ environmentDigest:input.environmentDigest,envelopeDigest:input.envelopeDigest,
+ taskId:input.taskId,appliedBaselineDigest:input.baselineDigest,
+ appliedCandidateDigest:input.candidateDigest,
+ baselinePassed:passes(input.baseline),candidatePassed:passes(input.candidate),costUsd:0};
+};
+const options={tenantId:'release',projectId:'acceptance',environment:{version:1},
+ envelope:{network:false,promotion:false},baseline,candidate,
+ developmentTaskIds:['development:0'],confirmationTaskIds:Array.from({length:20},(_,i)=>'confirmation:'+i),
+ policy:{minCases:20,minWins:5,maxRegressions:0,minGain:0.05,alpha:0.025},
+ budget:{maxUsd:0,maxCalls:20,pairUsd:0,timeoutMs:1000},
+ evaluator:{surfaceAware:true,evaluatePair}};
+const result=await confirmContrastive(options);
+assert.equal(calls,20);assert.equal(result.verdict,'confirmed');
+assert.equal(result.stats.wins,6);assert.equal(result.stats.pValue,0.015625);
+assert.equal(result.promotion,'not-authorized');
+assert.equal(pairedSignPValue(5,0),0.03125);
+const invalid=await confirmContrastive({...options,evaluator:{surfaceAware:true,
+ evaluatePair:async input=>({...await evaluatePair(input),envelopeDigest:'wrong'})}});
+assert.equal(invalid.verdict,'inconclusive');assert.equal(invalid.stats.cases,0);
+console.log(JSON.stringify({installedApi:'passed',cases:20,pValue:result.stats.pValue,
+ provenanceMismatch:'rejected',promotion:result.promotion}));
-Original file line number
+Diff line change
@@ Expand Up @@
     ### New
+    * Bounded contrastive harness development: `@metaharness/darwin/contrastive` adds scoped proposals, actual probes, selective composition and metered experiment memory behind immutable caller policy. [API guide](packages/darwin-mode/CONTRASTIVE.md). Independent paired confirmation adds unseen-task, statistical and regression checks. Public intent-classification evidence remains separate from general agent performance claims.
     [Watch Darwin evolve a harness](docs/visual-walkthrough.md#evolve) · [Explore model routing](docs/visual-walkthrough.md#routing) · [Review experimental extensions](docs/visual-walkthrough.md#extensions)
     - **Run an experimental ARC-AGI-3 harness from the ChatGPT UI.**
@@ Expand Down @@