{{ message }}
Repository navigation
Conversation
Includes source, built distribution, scoped memory, live editor adapter, adversarial tests, synthetic benchmark, and release evidence. Validated on ruOS. Published tree exactly matches local reviewed tree 70a8e0f (local source commit 7f9455f, evidence commit d92e932).
The existing CLI dependency is ^0.10.0 and cannot use the new 0.11.0 workspace package. Regenerate the lockfile to retain its published 0.10.3 dependency. Clean npm ci and Darwin build pass. Published Darwin tarball is unchanged.
…evidence Add frozen paired holdout validation, exact statistical gates, total deadline enforcement, conservative budgets and bound receipts. Include 99 confirmation tests and a pinned CLINC150 experiment. The live candidate tied baseline at 38/40 and cost more; both superiority gates rejected. 830 tests passed, 19 existing skips; package build, fresh consumer acceptance and production audit passed. Validated ruOS source tree matches local commit 1eb56a8d55f2ff430a4095cc1f10a3f266a04f49 exactly. Rust CI blocker tracked in #408.
ruvnet
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Darwin 0.12.0: bounded contrastive development and independent confirmation
Development gains can disappear on new tasks. This PR adds an opt-in contrastive development API and a separate confirmation API that evaluates one frozen candidate on disjoint paired tasks before a caller considers adoption.
The development API supports scoped invariant/domain edits, measured composition, deduplicated probes, effect budgets, execution digests and scoped memory. Confirmation adds frozen policies, exact paired statistical tests, minimum gain and regression gates, all-cases completion, one total deadline, conservative reservations and receipts bound to execution context. Existing evolve APIs remain compatible. Neither API authorizes deployment.
Evidence
830 tests passed with 19 existing skips, including 99 focused confirmation tests. Clean installation, TypeScript compilation, deterministic adapter checks, fresh tarball consumer acceptance and production npm audit passed. The production audit reported zero vulnerabilities at audit time. Security assessment and limits are in packages/darwin-mode/bench/confirmation/security-review.md.
One frozen public CLINC150 pilot used 24 development and 40 independent confirmation requests across eight finance intents. Development improved 23/24 to 24/24, but baseline, candidate and control each scored 38/40 on confirmation. Both superiority comparisons had zero paired wins/losses and p = 1; both rejected. Candidate inference cost increased 15.68%. Keep the baseline.
The pilot made 253 provider requests, reporting $0.007642223 usage charges. Conservative reservations totaled $0.596 under a $1 client cap. No unresolved requests. This small public subset does not establish general accuracy or agent performance gains. No holdout tuning or reruns were performed.
The preceding 0.11.0 synthetic benchmark and editor smoke remain in docs/evidence/contrastive-0.11.0. They test mechanisms and integration, not general model performance. New public-pilot evidence and validation are in docs/evidence/confirmation-0.12.0.
Publication and CI
GitHub release: https://github.com/ruvnet/metaharness/releases/tag/darwin-v0.12.0
Package: @metaharness/darwin@0.12.0, subpath @metaharness/darwin/contrastive.
Source commit ee4ea2b has exactly the ruOS-validated tree 51cdb2ea253b3c9f8ebef3757e3364bad89e6b19. Full frozen manifests, candidate documents, results and exact tarball are attached to the release.
This PR remains unmerged pending required CI. Existing Rust dependency resolution is blocked by Candle 0.8 and yanked zip 1.x, tracked in #408. No Rust CI or cargo audit success is claimed. The CLI's existing Darwin ^0.10.0 dependency retains its separate locked 0.10.3 package; clean npm ci passes.