Repository navigation
feat(simulate): version agents, environments and scenario suites and pin runs to them - #3533
JayaSurya-27 wants to merge 39 commits into
Conversation
…oning # Conflicts: # api_contracts/openapi/management-api-contract-debt.generated.json # api_contracts/openapi/runtime-management-api-contract-debt.generated.json # frontend/src/api/contracts/api-surface.generated.js # frontend/src/api/contracts/openapi-contract.generated.js # futureagi/mcp_server/catalog/tools.generated.json
…oning # Conflicts: # api_contracts/openapi/management-api-contract-debt.generated.json # api_contracts/openapi/runtime-management-api-contract-debt.generated.json # frontend/src/api/contracts/api-surface.generated.js # frontend/src/api/contracts/openapi-contract.generated.js # frontend/src/sections/simulate/environments/workspace/workspace.constants.js # futureagi/mcp_server/catalog/tools.generated.json # futureagi/simulate/services/hosted_harness_conversation.py
- The create request carries the agent's variable values; they are kept only in the agent version. - Each environment gets its own agent version, pinned for its runs and builds. - Environments on the same provider assistant share one agent definition. - The build's resolved commit is written onto the environment's own version. - The launcher sends the values with the create request instead of minting vault refs.
- Watch each environment's GitHub source through a signed webhook and a scheduled poller, with per-owner, per-org and hourly budgets. - A new commit, an uploaded archive or a prompt edit creates an agent version that is checked, then offered to each environment that uses it. - Discards and roll backs hold per environment; the first sight of an existing environment records its head instead of creating a version. - Sources are tracked by GitHub repository id; private repositories are refused at creation. - The versions panel shows each agent's source and lets people upload new source or edit the prompt.
- New compare endpoint pairs scenarios by key and reports each one as improved, regressed, no change or inconclusive from the pass rates over trials. - Scenarios only in one run, and scenarios whose sub-goals changed, are listed separately. - The response names the agent, environment and scenario versions each run used. - Run analytics gets a picker to choose the run to compare against.
…nges - A new default agent version, or a newer version an environment can pick, raises a nudge card in the builder chat with what changed: files, prompt, settings and variable names, never values. - Moving the environment default raises a scenarios nudge; nudges are re-checked when a build or extend completes and closed when the environment is deleted. - One environment nudge is pending at a time: the default's own card wins, and only the newest version is offered. - GitHub compares use their own capped share of the hourly budget, and patches are redacted against the agent's credential values. - The nudges list and Skip endpoints, the chat card, the Agents panel action and the unread dot; the old fixture banners are removed.
- With versioning on, chat edits, scenario edits and Extend write one draft per piece instead of changing the live environment. - One Save turns every unsaved piece into new versions and makes them the defaults, which raises the same nudges as a manual move; Discard drops them. - Out-of-date drafts are refused and replaced by the next edit; Save and Discard wait while the chat is working, but not on an unanswered question. - The Scenarios tab marks unsaved rows, which can't be run until saved, and runs use the saved suite; the Contract and Overview views stay on the live version. - Chat checkpoints are packed deterministically so the same checkpoint is not saved twice.
- Remove the unused authoring activity, the unmounted fixture agent screens, the fixture versions and the environment state's version slice. - The old create page sends credential values with the create request; secret-values/ now always needs an environment and creates a new agent version. - Chat saves always write a new content-keyed archive instead of overwriting in place. - Document the three versioning flags (all off by default) and the backfill-then-flag upgrade order.
- The e2e stack turns on versioning, nudges and certify, blanks sandbox-provider keys, and sets a test encryption key. - Seeds create a backfilled environment, apply edits and plant a finished run, scoped to the flow's org and workspace. - Flows SIM-E2E-001 to 004 cover the versions tab, run picks with warnings, roll back, and a credential change that makes a new agent version. - Seeding refuses to run when a real sandbox-provider key reaches the stack.
- Rebuild queues a builder turn with a names-only briefing, then lands its result as new environment or scenario versions, reports nothing to change, or fails and puts the person's unsaved edits back. - Unsaved edits are held aside during the turn and restored on every outcome that doesn't land; Save, Discard and default moves wait while a rebuild runs. - Stop works from the card, the agents panel and the composer; a tick closes rebuilds that never start or stall, and retries deferred activations fairly across organizations. - Results are redacted against every agent the turn could see, and only proofs made after the turn started count.
The e2e harness, seeds and flows for environment versioning run locally only and are not part of this branch.
JayaSurya-27
changed the title
hadarishav
left a comment
hadarishav
left a comment
There was a problem hiding this comment.
Requesting changes for three confirmed bugs in the platform implementation.
The review and reproductions used b0b663241be69de5e0a6f3e6129289bf30c3e49c. Before posting, I checked the current head 183622f1dc985b9a4cbb42cf4d25b1b9d54638bc: all three files below are unchanged. This is not a complete review of the newer commits or all 274 changed files.
-
P1: Save can publish a new environment version that still executes the old world.
drafts._save_pairaccepts a carried bundle and_composed_pairmarks the new pair sealed without establishing that the bundle contains the edits. The gateway copies that bundle directly for simulation-only execution. A database-backed reproduction saved changed world bytes, created a new environment version, and left the executable bundle containing the old world. Require a bundle certified for the exact saved pair; otherwise prepare it through--certify-only. Checking the draft hashes alone is insufficient whenworkspace_sealedcontains a carried bundle. -
P1: Real world-data changes are excluded from environment identity.
hashing.pyexcludes all ofgeneric-harness/world-ir.json, although ALK preferentially uses it as runtime seed data when present. Two valid worlds with balances 10 versus 999 produce different packaged runtime data but the same environment hash. A database-backedrecord_draftreproduction then produces no environment draft and retains the previous archive. Hash the semantic world contents, excluding only metadata that must not affect identity. -
P2: Equal observed pass rates incorrectly imply “No meaningful change.”
run_compare.verdictreturns this label before checking uncertainty whenever the rates match. For 1/1 versus 1/1, the actual function reports that label with an interval of approximately -79 to +79 percentage points. PRD §4.10.7, lines 552–555 requires the entire interval inside the equivalence margin. Remove the shortcut and update the test that currently enforces it; wide intervals should remain inconclusive.
Verification on the reviewed commit: 1,301 existing backend tests passed, plus two database-backed reproductions confirming the integrity bugs. Four phone-agent tests needed a test-only outbound caller ID and passed on rerun. Passing tests do not resolve the findings above.
Browser and live E2B/provider execution were not checked. Review remains incomplete; fixing these findings is not an approval of the whole PR.
|
The implementation is a substantial versioning foundation, but fixing the reported bugs does not complete the agreed plan. Please resolve the scope of each item below before presenting this work as complete. For each row, either implement the requirement or explicitly defer it in both this PR description and the relevant Linear ticket, with Chintan's alignment recorded. A deferral should identify what is excluded, the follow-up ticket and owner, and the user-visible limitation. Please distinguish the current milestone from the full feature, including what “Fixed TH-7948” covers. TH-8656 and TH-8657 were still Todo when reviewed; these are scope decisions, not a demand to silently absorb every related ticket into this PR. The references below use the exact PRD revision reviewed. Linear references use requirement numbers because tickets have no stable line numbers. The table describes gaps identified at reviewed head
Please split this work into stacked PRs for appropriate review. The reviewed platform diff contains 274 files, 80,680 additions and 6,174 deletions. The review so far covered selected critical paths, not every changed line. Four confirmed bugs across the pair are not evidence that only four exist. A suggested stack is:
Adjust the boundaries to preserve coherent, testable behavior. Each PR should show only its incremental diff against its predecessor, map to the agreed requirements, include focused verification, and document deployment dependencies. Generated contracts and tests should accompany the behavior they cover. Please provide the stack order and a coverage checklist so reviewers can track what has actually been reviewed. Current verdict remains request changes and review incomplete. Fixing the listed bugs alone is not approval of the full implementation. Browser and live E2B/provider flows remain unverified. |
A Save could make the default a pair whose carried or chat bundle was sealed for another world or suite, so runs replayed the old world. The chat's bundle is used only when it was resealed for exactly the saved pair (decided by the bundle manifest, not the object key), the default's bundle is carried only when the environment is unchanged and the suite's certify hash is equal, and anything else is left for the first run to certify. An environment without an agent to certify for is told to make the change in chat.
world-ir.json stays out of the environment hash; the tests pin that a world.sqlite change moves the hash and that world.sqlite lands in the environment piece.
A validation repair is now written into world.sqlite, so it versions the environment; a world-ir.json-only change keeps the version.
|
Thanks for the review and the reproductions. Pushed to 1. Save publishing the old world: fixed in
Tests, all in
The fixtures for chat saves and rebuild landings in the drafts, record, rebuild and runs tests now model the real chat reseal; previously they carried the default's manifest unchanged. The rollback tests now save their Scenarios-tab edit with bundle preparation on. Two gaps remain, both marked in the code or limited to old data:
Local E2E on the full stack, with a sandbox template built from the ALK fix branch:
2. World data and the environment hash: no change to excluding That file embeds the fingerprint of the agent's source model, and its row ids and value coercion depend on the agent's source model. Hashing it would tie environment identity to the agent version. The world data is identified by the hashed world files ( The fixes below keep
What the code now guarantees: for an environment whose authoring tree holds Your reproduction (a World IR of 10 vs 999 beside the same Four exceptions remain:
ALK tests for this guarantee:
Local E2E: on an environment whose stored 3. Compare verdict: deferred with compare as a whole. The compare UI is hidden in this PR ( Updated after the second review round: a chat bundle is trusted only when ALK's stamp matches the saved tree and a failed reseal fails the turn ( |
|
Thanks for mapping the scope this precisely. Here is where each item stands. The PR description now says "Part of TH-7948"; this PR doesn't close it.
Compare as a whole is hidden in this PR ( Stacked PRs: we are keeping this as one PR. Many people work on
Live E2B:
Updated after the second review round: dev already writes |
hadarishav
left a comment
hadarishav
left a comment
There was a problem hiding this comment.
Focused re-review of the fixes. The normal stale-bundle carry path is improved, but the unpromoted-reseal case below still allows a saved environment to execute an older world. Please address it before resolving the original bundle-consistency finding.
Validation: 202 focused platform tests passed against an isolated PostgreSQL database with migrations skipped. A separate reproduction using a real ALK-generated bundle failed on the mismatch below. Hosted/E2B calls were not rerun.
The feature deferrals and current PR structure are accepted for this review. This is not approval of the entire PR.
| except (S3Error, HostedHarnessError, ValueError): | ||
| return False | ||
| held = default.get(SEALED_MANIFEST) | ||
| return held is None or archives.sha256_hex(held) != manifest |
There was a problem hiding this comment.
[P1] An unknown manifest does not prove that the bundle was resealed for the current content
This returns true for any manifest not present in the current drafts/default. record_draft() then stamps the current environment/suite hashes onto it, and _save_pair() trusts that association. A bundle from an earlier, unpromoted reseal can therefore be saved for a later world.
The failure path is reachable across the chat lifecycle: ALK can reseal and then fail to publish, leaving the bundle in a warm sandbox. A later turn edits the world but fails to reseal. _publish_if_changed() reports a warning, but _handle_command() still checkpoints and emits outcome: success, which the platform schedules for promotion. The earlier manifest is unfamiliar here, so it is classified as newly resealed for the later edit.
I reproduced the platform save with a real ALK-generated, validated bundle: the saved authoring SQLite contained second-edit, its sealed seed contained first-edit, and default_pair_sealed was true. A separate ALK reproduction confirmed a failed reseal still emits a promotable success event. Ordinary explicit discard retires the sandbox; this finding concerns unpromoted reseals.
Please bind the bundle to the authoring content it actually sealed, or conservatively require certification when that association cannot be verified. Also prevent a failed reseal from being promoted through the successful-turn fallback.
There was a problem hiding this comment.
Fixed in 038acb5b8 (platform) and future-agi/agent-learning-kit d3d153dc.
- ALK now stamps a resealed bundle's manifest with
metadata.sealed_from: a digest of the authoring files the reseal carries (contract.json,sub_goals.json,world.sqlite,scenarios/). A reseal that cannot carry the world leaves no stamp. _resealedtrusts a chat's bundle only when that stamp matches the tree being saved and the bundle's other environment files are unchanged from the default's. A bundle from an earlier, unpromoted reseal carries the earlier world's stamp, so it is not trusted: the pair is saved unsealed and its first run certifies it.- A chat turn whose reseal fails now ends with
outcome: failedthrough the same path as other failed turns, so its checkpoint is not promoted.
Tests:
- Platform:
test_harness_versioning_reseal.py, includingtest_a_bundle_stamped_for_an_earlier_world_is_not_resealed,test_an_unstamped_bundle_is_not_resealedandtest_a_bundle_resealed_beside_an_edit_it_does_not_carry_is_not_resealed. The drafts, record, rebuild and runs fixtures now model the stamp. - ALK:
test_a_failed_reseal_ends_the_turn_failed_with_nothing_to_land,test_a_bundle_is_stamped_only_when_a_reseal_carries_its_world,test_a_bundle_keeps_the_stamp_of_the_world_it_was_sealed_from.
Still open, marked # TODO: in _resealed: a contract.json-only edit still reuses the resealed bundle. The reseal copies contract.json, but not what a build derives from it. Treating any contract change as unsealed broke rebuilds that add a tool, so this needs a check against ALK first.
Local E2E: a chat world edit was saved unsealed (environment v15, suite v6; the chat's bundle was not reused), then the run certified the pair with --certify-only and completed 1/1 calls (run 59572bb5). The path where a stamped chat bundle is accepted was not reached live; the reseal tests cover it.
hadarishav
left a comment
hadarishav
left a comment
There was a problem hiding this comment.
Completed the broader review against the seven-section checklist at platform 234fb00788c9c4c402c97558701267015dde20d0 and ALK 2e3a4b87aa747e5d64246947f26b2b1559b9fca8. Four additional findings are attached inline. These are separate from the already-posted stale-bundle and SQLite-default findings. Approval remains withheld pending the correctness fixes.
| Checklist section | Coverage and result |
|---|---|
| 1. Models, hashing, archives, migrations/backfill | Inspected identity, constraints, archive validation, backfill safeguards and tenant filtering. No additional confirmed finding. |
| 2. Agent versions, credentials, preflight and detection | Traced creation, preflight, activation, secret resolution and GitHub observation. Confirmed pending credential rotation can be undone by a push. |
| 3. Drafts, Save/Discard and defaults | Traced save/discard, restoration and concurrent chat writes. Confirmed rebuild landing can activate a newer unchecked draft. |
| 4. ALK certification/proofs and platform assembly/pinning | Inspected proofs, bundle assembly/cache, selected versions, scenario overlays and launch. Previously posted correctness findings remain. |
| 5. Nudges and rebuild lifecycle | Traced dispatch, gates, checkpoint promotion, completion, retries and stop/recovery. Confirmed landing race and partial credential disclosure in report redaction. |
| 6. UI and APIs | Static review of version/draft/run controls and caller integration. Checked changed frontend modules for unresolved local imports and UI route templates against OpenAPI. No additional confirmed finding; no browser/Vitest run. |
| 7. Supporting changes | Inspected telemetry, schedules, flags, startup guards and rollout instructions. Confirmed documented Compose backfill write command is rejected. |
Verification: 1,695 backend test passes in the selected suites. Four phone-fixture failures passed after supplying the missing test-only caller-ID configuration. The broader ALK selection finished with 544 passes, three skips and three failures also reproduced on the base branch. Independent reproductions confirmed the findings; external preflight probes were mocked, and the rebuild race used a controlled database-backed interleaving through the real service paths.
This was a thorough section-based review, not exhaustive verification of every changed line or runtime flow. Frontend review was static; no browser/Vitest or live E2B/Vapi/Retell execution. Migrations were inspected but not executed against representative existing data; backend test schemas used --nomigrations. Generated contracts were checked structurally, not manually line by line. These limits mean passing tests are not evidence that these are the only remaining defects.
The accepted feature deferrals and current PR structure are not being reopened. After the fixes, the next verification priorities are migration/backfill against existing data and browser-to-hosted flows.
| if report is None: | ||
| close(state=State.FAILED, error={"code": "no_report"}) | ||
| return nudge | ||
| draft = _result_draft(locked, nudge) |
There was a problem hiding this comment.
[P1] Bind rebuild landing to the draft that passed checks
Checkpoint promotion and landing happen separately. land_nudge_result reads the passing report from the nudge checkpoint, then selects the latest fresh draft without matching its hash/revision to that checkpoint. A run-chat edit in between can overwrite the draft and be activated by the old report. A database-backed deterministic interleaving through the real run-chat compose/enqueue/promote path changed the draft from nudge revision 1 to chat revision 2; landing reported applied and made the unsaved chat change live. This reproduces with certification enabled and an unsealed save, independently of the previously posted stale-bundle issue. Bind the report to the checked draft hash/revision and refuse landing if it changed.
There was a problem hiding this comment.
Fixed in 9e23648e2. Before landing, land_nudge_result now checks whether another chat saved over the rebuild's result (_saved_over), and if so fails the landing with draft_changed:
- a fresh draft the rebuild didn't write, whose hash differs from the draft it set aside at the start; or
- with no draft left, a draft this turn's chat discarded by saving the default (scoped to the turn's start, set-aside drafts excluded).
restore_draft_snapshot no longer brings the turn-start draft back over that save.
Tests in test_harness_versioning_rebuild.py: test_a_run_chat_save_before_the_landing_is_not_landed[edit|unchanged], test_a_run_chat_save_of_the_default_before_the_landing_is_not_landed[default|draft], test_a_run_chat_save_before_a_landing_retry_is_not_landed, and test_unsaved_edits_the_rebuild_leaves_as_they_are_still_land. Both cases of test_a_run_chat_save_of_the_default_before_the_landing_is_not_landed fail on the old code.
Not reproduced in the local E2E: the window between checkpoint promotion and landing can't be hit through the API, so this is covered by tests only.
| environment.id, | ||
| ) | ||
| return "skipped_not_runnable", None | ||
| parent = environment.agent_version |
There was a problem hiding this comment.
[P2] Preserve pending credential rotations when observing a GitHub commit
_observe_one always creates the push version from environment.agent_version, ignoring a pending credential request. Reproduced A1 old key -> pending A2 rotated key -> GitHub observer creates A3 from A1 -> A2 passes/activates -> A3 passes/activates. The final pinned credential is the old key. UI restrictions cannot prevent background polling/webhooks from doing this. Carry forward the outstanding requested version or defer observation while it is pending. External preflight calls were mocked; database version creation, encryption/decryption, activation and pinned secret resolution were real.
There was a problem hiding this comment.
Fixed in bee5835ae. _observe_one now builds the push version on the outstanding requested version when there is one (_outstanding_request), so in your sequence A3 is built from A2 and keeps the rotated key. _outstanding_request also counts a change that passed while the environment was busy (environment_busy), which was the other way the old key could return.
Tests in test_harness_agent_changes.py: test_a_push_while_a_credential_change_is_checked_keeps_its_values[first|second] (either version activating first), test_a_push_on_a_failed_credential_change_is_asked_again_once_fixed, and test_a_push_after_a_credential_change_passed_while_busy_keeps_its_values.
Not run live: the push to the test repo was blocked on the local stack, so rotation plus push is covered by tests only.
| ), | ||
| ] | ||
| masked = _redacted( | ||
| environment, nudge, [text[:cap] for _part, _at, text, cap in secret_parts] |
There was a problem hiding this comment.
[P2] Redact rebuild diagnostics before truncating them
_report truncates strings before _redacted runs. A credential crossing the cutoff becomes a prefix that no longer matches the known full secret and survives into persisted summary/error fields exposed by nudge_item. Running the actual report and redaction functions with a fake credential showed normal redaction replacing it with [REDACTED], while report output retained FAKE-CREDENTIAL-123. Redact complete strings before limiting their displayed length. This confirms partial secret disclosure, not an entire usable credential.
There was a problem hiding this comment.
Fixed in 0f7848ad2 (platform) and future-agi/agent-learning-kit c393f7b3. _report now masks before it cuts. Each text is:
- normalised (control characters and ANSI escapes stripped);
- cut to its cap plus 16 KiB (
SECRET_VALUE_MAX, longer than any credential value), so a value across the cap is still whole and masking stays cheap; - masked;
- cut to its cap.
RESULT_FILE_MAX is now 4 MiB. On the ALK side, texts are kept up to 4000 characters and each list to 50 entries, so a value is not cut before the platform masks it.
Tests:
- Platform:
test_a_value_across_a_cap_is_masked_before_the_report_cuts_it,test_only_a_bounded_prefix_of_each_text_is_masked,test_control_characters_cannot_push_a_value_past_the_masked_prefix. - ALK:
test_texts_the_platform_masks_keep_a_value_whole_past_its_500_cap,test_the_result_keeps_only_what_the_platform_reads.
Covered by tests only; not exercised in the local E2E.
|
|
||
| ```bash | ||
| docker compose exec backend python manage.py harness_versions_backfill --check | ||
| docker compose exec backend python manage.py harness_versions_backfill |
There was a problem hiding this comment.
[P2] Authorize the documented backfill write command in Compose
Both supported Compose configurations set NO_STARTUP_DB_MUTATIONS=true. The documented --check command is allowed, but the following write command is rejected by model_hub/apps.py during Django startup. Verified the actual guard functions reject that invocation and accept explicit local mutation mode. For self-hosted Compose, include docker compose exec -e NO_STARTUP_DB_MUTATIONS=false backend python manage.py harness_versions_backfill (app for Standalone); hosted rollout should use its authorized operator workflow.
There was a problem hiding this comment.
Fixed in fa3f923c4. INSTALLATION.md now runs the write as a one-off operator process. The three variables apply to that process only; the running services keep NO_STARTUP_DB_MUTATIONS=true.
docker compose exec backend python manage.py harness_versions_backfill --check
docker compose exec -e SERVICE_TYPE=bootstrap -e STARTUP_DB_MUTATION_MODE=operator -e NO_STARTUP_DB_MUTATIONS=false backend python manage.py harness_versions_backfillUse app in Standalone. Helm has the same form through kubectl exec ... -- env SERVICE_TYPE=bootstrap STARTUP_DB_MUTATION_MODE=operator NO_STARTUP_DB_MUTATIONS=false python manage.py harness_versions_backfill. deploy/env-reference.toml points at that section.
Test: model_hub/tests/test_clickhouse_cache_warm.py::test_the_documented_harness_backfill_run_is_allowed[local|production].
Local E2E:
- the plain write command was refused by the startup guard, as you found;
--checkreported 6 environments current, 0 errors;- the documented operator form ran (
current 6).
KarthikAvinashFI
left a comment
KarthikAvinashFI
left a comment
There was a problem hiding this comment.
Backend architecture review (frontend and tests not reviewed here). The model follows the PRD's decoupling and reuses the existing pieces well (AgentVersion side table, build_run_comparison extended, canonical_json moved, HostedSourceAcquirer reused). Requesting changes for one architectural issue plus a few duplicates, and the branch needs latest dev (402 behind, 22 conflicting files, migration 0098 collides with dev's 0098_agent_version_setup).
- Identity is re-implemented from ALK's layout:
tree_digestis a "byte-for-byte port" of ALK's_artifact_digest(hashing.py:133), and which ALK files count is hard-coded inhashing.py:50-69andarchives.py:37-60. Every new file ALK writes changes identity until it is listed here; the world-hash bug was this drift. ALK already writesworld_ir_hashandscenario_set_hashinto its certificate; can the platform store identities ALK reports instead of re-deriving them? Worth checkinggeneric-harness/setup-checks-passed.json, which is on none of the lists. run_compare.wilson_interval(run_compare.py:44) duplicatesrun_reliability_v3._wilson(run_reliability_v3.py:16); one helper for both.compareinviews/harness_environment.py:508-581resolves runs and applies status rules in the view; please move that to a service.- Migrations
0100to0102alter tables created in0098in this same PR; fold them into one when renumbering on top of dev. devnow hasAgentVersionSetup(simulate/models/agent_setup.py:9), and the plan detects provider changes by itssetup_hash, but none of this PR's backend uses it; after mergingdev, provider-change detection should read it instead of keeping a second record.- The removal of the
author_hosted_harness_jobactivity is unrelated to versioning; it belongs in its own small PR.
Note on this review. Given the size of the diff, this review was done with AI assistance and I have not yet gone through it line by line myself, so please treat it as a first pass. It is a code and architecture review of the backend at 234fb007 against the PRD, the plan and our backend standards. The frontend and the test files were not reviewed, and nothing was run: no tests, no frontend, no builds and no live runs. Each finding cites the line it was read from, but context outside the diff may be missed, and I will confirm or drop points as we discuss them.
| def tree_digest( | ||
| files: Path | Files, *, exclude: Callable[[str], bool] | None = None | ||
| ) -> str: | ||
| """Byte-for-byte port of ALK ``authoring_runtime_validation._artifact_digest``. |
There was a problem hiding this comment.
A byte-for-byte port of ALK's digest means the two must change together forever. ALK already writes world_ir_hash and scenario_set_hash into its certificate; can the platform store identities ALK reports instead of re-deriving them?
There was a problem hiding this comment.
Fixed in part through 32df8c74c (see the reply on hashing.py:51). The unhashed list in hashing.py now names only root-level files. archives.py still lists which generic-harness/ records travel with the bundle, and that list gained two entries this round.
On storing what ALK reports:
- ALK writes
world_ir_hash,scenario_set_hashandcontract_hashonly when runtime validation certifies. Chat edits and Scenarios-tab saves are versioned before any certify, so for them the certificate is stale or absent. - They also measure different things:
world_ir_hashembeds the source-model fingerprint, so the agent's source digest. Every agent commit would mint a new environment version.scenario_set_hashcoverschecks/and__pycache__.contract_hashcovers raw contract bytes, includingchosen_evalsand the provider prompt, which the platform leaves out on purpose.
tree_digest keeps ALK's byte layout only so both sides compute the same digest for the same tree. It does not decide which files count; the lists do.
A follow-up could have ALK write a manifest of identity files, with the platform hashing exactly those. That would remove the lists.
| # In an authoring tree but not part of the environment's identity: the contract is | ||
| # hashed separately; the rest embed the agent's source digest, belong to the suite or | ||
| # to a build, or are run noise. | ||
| ENVIRONMENT_UNHASHED_FILES = frozenset( |
There was a problem hiding this comment.
These lists (and the ones in archives.py:37-60) hard-code ALK's workspace layout, so every new file ALK writes changes identity until it is listed here; the world-hash bug was this drift. Worth checking generic-harness/setup-checks-passed.json, which is on none of them.
There was a problem hiding this comment.
Confirmed, and fixed in 32df8c74c. generic-harness/setup-checks-passed.json is runtime validation's resume cache: a list of keys over the bundle, the invariants and each scenario's folder. Any validation pass, even for a suite-only change, rewrote it, and it landed on the environment side and was hashed, so it minted a new environment version.
What changed:
hashing.py: the wholegeneric-harness/root is now outside environment identity. ALK keeps only derived records and caches there, and its own candidate hash skipsgeneric-harness/too. A file ALK adds there later can no longer move the hash.archives.py:setup-checks-passed.jsonandruntime-plan-repair-*are sealed, so they travel with the bundle they describe.provider-import-profile.json(the provider agent's fetched definition) is no longer stored in a version, so a provider prompt edit no longer changes the environment hash.source-data-review-early-error.txtis dropped as a log.
- Tests:
test_harness_versioning_hashing.py::test_ignores_build_outputs_suite_files_and_noise(the cache file and a futuregeneric-harness/file) andtest_harness_versioning_archives.py(sealed and dropped sets).
Side effect: trees built on this branch hold the cache file, so their recomputed hash changes once. The first chat save after this change mints one new environment version. Local E2E showed this: a save with no world edit moved environment v13 to v14, and its change summary counted two world files changed. Versioning is off on dev and the backfill runs after deploy with the new rule, so this only affects environments versioned from this branch.
| _UNRUN_OUTCOMES = ("queued", "in_progress", "error") | ||
|
|
||
|
|
||
| def wilson_interval(passed: int, trials: int) -> tuple[float, float]: |
There was a problem hiding this comment.
This duplicates run_reliability_v3._wilson (run_reliability_v3.py:16); one helper for both.
There was a problem hiding this comment.
Done in 44ee13ade. run_compare.wilson_interval now calls run_reliability_v3.wilson (renamed from _wilson). It takes an optional z so compare keeps its exact z; with 1.96, one of Newcombe's published values differs in the fourth decimal. No test change; the existing compare and reliability tests pass unchanged.
| reject_unknown_fields=True, | ||
| ) | ||
| @action(detail=True, methods=["get"], url_path="compare") | ||
| def compare(self, request, pk=None): |
There was a problem hiding this comment.
This resolves runs and applies status rules in the view; please move that into a service and keep the view to validate, call, respond.
There was a problem hiding this comment.
Agreed, no code change in this PR. The compare endpoint and UI are hidden here (RUN_COMPARE_ENABLED = false), and compare as a whole moves to TH-8657, which waits on the designs and flows. Its response contract changes there, so the run resolution and status rules will move into run_compare then, rather than being moved twice.
| from django.db import migrations, models | ||
|
|
||
|
|
||
| class Migration(migrations.Migration): |
There was a problem hiding this comment.
0100 to 0102 only alter tables created in 0098 in this same PR; fold them into one migration when renumbering on top of dev (where 0098 is now taken).
There was a problem hiding this comment.
Agreed. 0098 to 0102 have to be renumbered on top of dev's 0098_agent_version_setup anyway, and they have not shipped anywhere, so folding them into one migration after dev's latest is safe. Merging dev and the fold are pending the author's decision and will be handled in a follow-up push.
|
@KarthikAvinashFI, replies to the review-body points that have no inline thread. The inline replies cover the rest. Fixes from this round are pushed to Merge latest #5, #6, the |
|
@hadarishav, thanks for the second round. All six findings are answered inline. Platform is now at Full platform and ALK suites were run on both heads. Every failure in them also fails on the pre-fix base. Local E2E on the full stack, driven through the platform API against hosted E2B sandboxes:
The certify runs on a sandbox template built from the latest ALK head timed out on one scenario's ready check, while the same pairs certified and ran on the previous template. That points at the template build. The release template will be rebuilt and verified before versioning is enabled. |
Summary
Adds versioning to simulation environments. An environment now has three versioned pieces: the agent (source, prompt and credential set), the environment (contract, world and sub-goals) and the scenario suite. Every run is pinned to one version of each. People can see every version and what changed in it, pick versions when starting a run (with warnings that never block), and roll back the environment or suite default. Agent changes (a GitHub push, an uploaded folder, a prompt or credential edit) become new agent versions that are checked and then offered to each environment through a nudge in the builder chat; answering yes rebuilds the environment as a new version. Builder edits are kept as unsaved drafts until Save. All three flags are off by default, so nothing changes on
devuntil the backfill has run andHARNESS_VERSIONING_ENABLEDis turned on.The companion agent-learning-kit PR (
feat/environment-versioning->dev) must deploy with this one: future-agi/agent-learning-kit#180Linked issues
Linear:
Part of TH-7948
AI use
AI use:
Type of change
E2E coverage
E2E: exempt (harness-gap: the environment versioning flows are kept out of this PR)
1) What changes were done
With
HARNESS_VERSIONING_ENABLEDoff, users see no change. With it on, the workspace gains a Versions tab, a run dialog, a roll-back menu, nudge cards, and a Save / Discard bar. The API adds the endpoints in the table below, adds pinned versions to run responses, and changessecret-values/(see Backward compatibility). Five migrations; no data migration.A. Tables and migrations —
simulate/models/harness_versioning.py,simulate/models/hosted_harness.pysimulate_harness_environment_versionsimulate_harness_suite_versioncertify_hash,scenario_count,authored_againstsimulate_harness_suite_memberscenario_keysimulate_harness_suite_validationsimulate_harness_agent_version_runtimeAgentVersion: source, prompt hash, encrypted credential blob, credential names and fingerprints, preflight status and reportsimulate_harness_nudgesimulate_harness_assembled_bundleHostedHarnessJobgainsagent_version,environment_version,suite_versionandassembled_bundleFKs (allSET_NULL, nullable). On an environment job they are the defaults; on a run job they are the run's pins.HostedHarnessConversationLeasegainsagent_version(the version whose source the sandbox holds).AgentDefinition.create_versionaccepts an optionalversion_number.0098_harness_versioning_tables— the seven tables0099_harness_versioning_pins— the job and lease FKs0100_harness_agent_runtime_hash_scope— rescopes the agent runtime hash unique constraint0101_harness_agent_runtime_check_started_at—check_started_aton the agent runtime row0102_harness_nudge_turn—command_messageandclosed_aton the nudgeB. Versioning core —
simulate/services/harness_versioning/hashing.py: content identity for environment, suite and agent;assembly_keyfor a version combination.simulate/services/canonical_json.pyholds the shared canonical JSON digest.archives.py: splits an authoring tree into environment and suite archives, composes them back, and stores them under content-addressed keys in the object store. No git is used for versions.versions.py:record_authoringrecords environment and suite versions from an authoring archive (identical content reuses the existing version);move_defaultandset_defaultsmove defaults with a conditional update, so concurrent moves get a conflict instead of overwriting.diffs.py: what changed between two environment, suite or agent versions (contract fields, tools, scenarios added / removed / changed).read.py: the version lists, a diff, suite members andrun_version_labelsfor run responses;require_versioningreturns 404 while the flag is off.C. Backfill —
simulate/management/commands/harness_versions_backfill.py,harness_versioning/backfill.py,backfill_check.py--checkis a read-only dry pass (rolled-back, read-only transactions; output carries ids, counts and codes, never scenario text, prompts or values). Also--job <id>(repeatable),--json,--output <file>.model_hub/apps.pyadds the command toOPERATOR_STARTUP_MUTATION_COMMANDSand lets the exact--checkform through as read-only.INSTALLATION.md):docker compose exec -e SERVICE_TYPE=bootstrap -e STARTUP_DB_MUTATION_MODE=operator -e NO_STARTUP_DB_MUTATIONS=false backend python manage.py harness_versions_backfill(appin Standalone), or on Helmkubectl -n futureagi exec deploy/futureagi-backend -c backend -- env SERVICE_TYPE=bootstrap STARTUP_DB_MUTATION_MODE=operator NO_STARTUP_DB_MUTATIONS=false python manage.py harness_versions_backfill. The running services keepNO_STARTUP_DB_MUTATIONS=true.D. Agent versions and credentials —
simulate/services/harness_agent_versions.py,harness_credentials.py,views/harness_job.pycredential_values); they are stored encrypted only on the agent version's side row. Everything else (diffs, reports, logs, task inputs) carries names and keyed fingerprints.preflight_harness_agent_version(simulate/tasks/harness_agent_versions.py) before it can become a default. Failed or stuck checks can be retried or discarded.secret-values/now requiresenvironment_id, creates the environment's next agent version from the new values and returns 202 with that version.harness_credentials.pyresolves names-only refs (harness-agent-version:<name>) from the pinned version's credential set; existingplatform-vaultrefs keep resolving.E. Agent change detection —
simulate/services/harness_agent_changes.py,harness_github.py,views/harness_github_webhook.py,simulate/signals.pyapi/harness/github/webhook/or the 10-minutepoll_harness_github_headsschedule), an uploaded folder, a prompt edit or a settings edit creates the next agent version throughcreate_agent_version.post_savereceiver onAgentVersionhands new or activated versions toon_agent_version_savedafter commit.GITHUB_APP_WEBHOOK_SECRETis new; unset, the webhook is ignored.F. Nudges and rebuild —
harness_versioning/nudges.py,rebuild.py,simulate/tasks/harness_nudges.pytick_harness_nudgesschedule closes rebuilds that never start or stall, and retries deferred activations across organizations.draft_changedand the old draft is not restored over that save.G. Drafts and Save —
harness_versioning/drafts.pyin_place=True) inhosted_harness_gateway.pyis removed.H. Run pins, assembly and bundle cache —
harness_versioning/assembly.py,hosted_harness.py,hosted_harness_gateway.py,harness_scenarios.pyassembly_keyinsimulate_harness_assembled_bundle. A run of the defaults with no cached bundle runs as before, unless that default pair was made without a sealed bundle. A run pinned to other versions certifies its own bundle (whenHARNESS_BUNDLE_CERTIFY_ENABLEDis on) and fills the cache for the next run.run_results_v3*.py,views/run_results_v3.py,views/run_test.py) returnenvironment_version,suite_version,pinned_agent_versionandagent_differs; null for runs from before versioning.run_testexecutions acceptsummary=truefor a light list of ids, statuses and pins.I. Run dialog and warnings —
harness_versioning/run_warnings.py,workspace/run/RunDialog.jsxagent_version_id,environment_version_id,suite_version_id; an absent pick is the default.run-plan/previews the versions a run would use and its warnings, without writing. Warnings never block:variable_missing,provider_key_unavailable,credential_unavailable,nudge_open,suite_unproved,agent_differs,bundle_cache_failed,bundle_cache_miss.agent_preflight_failed,credential_blob_too_large,version_not_runnable) instead of a generic error.J. Roll back —
versions.py(move_default),EnvVersionPin.jsx,MakeDefaultConfirm.jsxexpected_default_id; a stale one is refused. New runs use the new default; the previous version stays active.K. Run compare —
simulate/services/run_compare.py,run_results_v3_queries.py,runs/summary/RunCompare.jsxcompare/pairs two finished runs of the same environment by scenario key and reports each scenario as improved, regressed, no change or inconclusive, from pass rates over trials (Newcombe 95% interval against a ±10-point margin). Eval scores are context only.RUN_COMPARE_ENABLED); thecompare/endpoint is unchanged.L. Legacy retirement
author_hosted_harness_jobTemporal activity (temporal/activities/hosted_harness_gateway.py,tfc/temporal/common/registry.py).workspace/agents/(AgentsPanel,AgentCard,AddAgentDrawer,connect/*,PromoteDialog,DivergenceBanner, ...), the fixture versions (_fixtures/versions.js,_fixtures/envConfig.js),AgentRefreshBanner.jsxand the version slice ofstore/envState.js.HarnessCreate.jsx,prepareSourceForBuild.js) sends credential values with the create request instead of minting vault refs first.M. Frontend —
frontend/src/api/simulate-environments/versions.js,compareRuns.js,nudgeActions.js;frontend/src/sections/simulate/environments/workspace/versions/VersionsPanel.jsx,VersionDiff.jsx,VersionHeader.jsx): the three version lists, what changed in each, Make default. Shown only whenenv.versioning.enabled.agents/AgentVersionsPanel.jsx,AgentCredentials.jsx,AgentSourceChange.jsx,PreflightStatus.jsx,RebuildAction.jsx): source, check status, upload new source or edit the prompt, change credential values (names shown, values never echoed).buildEnvironment/console/NudgeCard.jsx: nudge card in the builder chat with Rebuild / Skip / Stop.drafts/DraftBar.jsx: Save / Discard bar with a change count.run/RunDialog.jsx,AgentDiffersTag.jsx: version pickers and warnings; pinned versions inRunDetail.jsxandSummaryTable.jsx.api_contracts/openapi/swagger.json,frontend/src/generated/api-contracts/*,futureagi/mcp_server/catalog/tools.generated.json).N. Review fixes (
661b0e336,4048014ff,234fb0078; second round038acb5b8…44ee13ade)drafts.py_save_pair,_resealed). A Save used to carry the chat's or the default's sealed bundle even when it was sealed for another world or suite, so a run at the defaults replayed the old world. Now:sealed_from); the platform trusts the bundle only when that stamp matches the saved tree and the bundle's other environment files are unchanged (038acb5b8). A bundle from an earlier, unpromoted reseal is not trusted, and a chat turn whose reseal fails now ends failed and is not promoted.contract.json-only edit still reuses the resealed bundle, since the reseal does not refresh what a build derives from the contract (# TODO:in_resealed).certify_hashis equal, since a run lays only overlay values (background noise, max turns, persona) over a replayed bundle.draft_not_runnable. An environment with no agent to certify for is told to make the change in chat.world.sqlite.generic-harness/world-ir.jsonstays out of the environment hash: it embeds the agent's source digest, and its row ids and value coercion come from the agent's source model. Tests pin that aworld.sqlitechange moves the hash and makes a new version, and that aworld-ir.json-only change does not. The harness side keeps the IR derived fromworld.sqlite(agent-learning-kit PR).generic-harness/is out of environment identity (32df8c74c). The harness keeps only derived records and caches there; its runtime-validation cache (setup-checks-passed.json) was hashed and minted a version on every validation pass. That cache andruntime-plan-repair-*are now sealed;provider-import-profile.jsonandsource-data-review-early-error.txtare no longer stored. Environments versioned on this branch get one extra environment version on their first save after this change.9e23648e2): a landing fails withdraft_changedwhen another chat saved over the rebuild's draft first, including a save of the default.bee5835ae): the push version is built on the outstanding requested version.0f7848ad2): texts are normalised, masked, then cut, so no prefix of a credential survives.fa3f923c4):INSTALLATION.mddocuments the operator form for Compose and Helm.2a7579508): a tool no sub-goal names is a warning, not a failure, on both sides.44ee13ade):run_compareusesrun_reliability_v3.wilson.world.sqlitecan't hold fails the attempt, and a rebuild may edit world checks uncited when the agent has no source (agent-learning-kit PR).O. Supporting changes
tfc/logging/sentry.py) and telemetry (tfc/telemetry/middleware.py) scrubenvironment_valuesandcredential_values; thesecret-values/,secret-files/and webhook paths are excluded from early body reads.simulate/apps.pyloadssimulate/signals.pyon startup; the GitHub poller lives insimulate/tasks/harness_agent_changes.py.INSTALLATION.md(backfill then flag),deploy/env-reference.toml(the three flags),futureagi/.env.example(GITHUB_APP_WEBHOOK_SECRET).New endpoints (on
harness-environments/{id}/,views/harness_environment.py)versions/versions/{piece}/{id}/diff/against)versions/{piece}/{id}/default/versions/agent/versions/agent/{id}/discard/,versions/agent/{id}/retry/versions/suite/{id}/members/drafts/,drafts/save/,drafts/{piece}/nudges/,nudges/{id}/answer/,nudges/{id}/stop/run-plan/compare/api/harness/github/webhook/2) Why the changes were done
simulate_harness_agent_version_runtime) instead of new columns on the agent tables. Readers decide by data (pins present or not), so pre-versioning environments and runs keep working.3) Tests written + scenarios each covers
Backend (
futureagi/simulate/tests/):Models, hashing, archives, recording
test_harness_versioning_models.py— table constraints and defaultstest_harness_versioning_hashing.py— environment, suite, agent and assembly hashes;world.sqliteis hashed,world-ir.jsonis nottest_harness_versioning_archives.py— split, compose and store archives;world.sqlitelands in the environment piecetest_harness_versioning_record.py— recording versions on authoring, reuse on identical content; aworld.sqlitechange makes a new environment versiontest_harness_versioning_history.py— run results read the pinned suite's contenttest_harness_amend_archive.py,test_harness_scenario_amend.py(additions) — scenario edits as versionsBackfill
test_harness_versioning_backfill.py— first versions and run pins for existing environments, idempotencytest_harness_versioning_backfill_check.py—--checkreports without writingAgent versions and changes
test_harness_agent_versions.py— credential sets, preflight, activation, discard and retrytest_harness_agent_creation.py— create with credential values, one agent version per environmenttest_harness_agent_provision.py— provisioning the pinned agent version in the sandboxtest_harness_agent_runs.py— runs and reruns with the pinned agent versiontest_harness_agent_source_api.py— upload source and prompt edit endpointstest_harness_agent_changes.py— new versions from pushes, uploads and edits; per-environment holdstest_harness_agent_receiver.py— theAgentVersionsave receivertest_harness_github.py— unauthenticated GitHub reads and the shared budgettest_harness_github_webhook.py— webhook signature, size and throttle checksNudges, drafts, rebuild
test_harness_nudges.py— raising, de-duplicating, redacting and closing nudgestest_harness_versioning_drafts.py— draft writes, staleness, Save and Discard;TestPairLadder: which bundle a Save keeps (chat world edit then a Scenarios-tab edit, a world edit whose reseal failed, backfilled and run-certified defaults, an earlier turn's bundle, overlay-only edits, an environment with no agent)test_harness_versioning_saves.py— every save of a versioned environment records versions, flag or no flagtest_harness_versioning_rebuild.py— rebuild turn: land, nothing to change, fail, stop, draft restore; a chat save before the landing is not landedtest_harness_versioning_reseal.py— which chat bundles count as resealed (stamp for this tree, an earlier world, unstamped, unreadable draft)test_harness_versioning_rebuild_report.py— report texts are masked before they are cutRuns, roll back, compare, API
test_harness_versioning_runs.py— run pinstest_harness_versioning_picks.py— run picks,run-plan/and warningstest_harness_versioning_assembly.py— assembly and the bundle cachetest_harness_versioning_rollback.py— default moves and conflictstest_harness_run_compare.py— scenario verdicts and thecompare/endpointtest_harness_versioning_api.py— the versions endpoints and the flag gateOther
test_agent_definition_services.py,test_harness_provider.py,test_hosted_harness_conversation.py,test_hosted_harness_eval_selection.pytfc/temporal/common/tests/test_registry.py,tfc/temporal/schedules/tests/test_register_schedules.py— activity registration and the two new schedulestfc/tests/test_sentry_filtering.py,tfc/deployment_telemetry/tests/test_privacy.py— credential values are scrubbedmodel_hub/tests/test_clickhouse_cache_warm.py— startup guard for the backfill commandFrontend (
frontend/src/;workspace/andbuildEnvironment/are undersections/simulate/environments/):api/simulate-environments/__tests__/versions.test.jsx,compareRuns.test.jsx,scenarioSelection.test.js— new API hooksworkspace/versions/__tests__/versionsPanel.test.jsx— Versions tab, diffs, Make defaultworkspace/agents/__tests__/agentVersionsPanel.test.jsx,agentCredentials.test.jsx,agentSourceChange.test.jsx— agent versions, credentials, source changesbuildEnvironment/console/__tests__/NudgeCard.test.jsx,NudgeCardRebuild.test.jsx— nudge card and rebuild statesworkspace/drafts/__tests__/DraftBar.test.jsx,workspace/scenarios/__tests__/unsavedScenarios.test.jsx— Save / Discard and unsaved rowsworkspace/run/__tests__/RunDialog.test.jsx,workspace/__tests__/EnvironmentWorkspace.runDialog.test.jsx— version picks and warningsRunAnalytics,RunDetail,RunsSummary,HarnessCreate,overview,header,panels,useEnvironmentsStoreand others in the diff); removed the tests of the deleted fixture agent screensFull backend and frontend suites were run clean before each step was pushed.
4) How to run / test
Backend tests
Frontend tests
cd frontend yarn test:run src/api/simulate-environments src/sections/simulate/environments src/pages/dashboard/harnessContract verification
Rollout
author_hosted_harness_jobactivity is removed).python manage.py migrate(0098–0102), thenpython manage.py register_temporal_schedules(addspoll_harness_github_headsandtick_harness_nudges).python manage.py harness_versions_backfill --check; fix every error it reports.docker compose exec -e SERVICE_TYPE=bootstrap -e STARTUP_DB_MUTATION_MODE=operator -e NO_STARTUP_DB_MUTATIONS=false backend python manage.py harness_versions_backfill(appin Standalone). On Helm:kubectl -n futureagi exec deploy/futureagi-backend -c backend -- env SERVICE_TYPE=bootstrap STARTUP_DB_MUTATION_MODE=operator NO_STARTUP_DB_MUTATIONS=false python manage.py harness_versions_backfill.HARNESS_VERSIONING_ENABLED=trueandHARNESS_BUNDLE_CERTIFY_ENABLED=truetogether.HARNESS_VERSIONING_NUDGES_ENABLED=true.GITHUB_APP_WEBHOOK_SECRETto accept push webhooks; without it, the poller still picks up pushes.HARNESS_VERSIONING_ENABLEDHARNESS_VERSIONING_NUDGES_ENABLEDHARNESS_BUNDLE_CERTIFY_ENABLEDUI steps (manual)
--check, then the operator form in Rollout step 4), turn the three flags on, and log in at http://localhost:3031.5) Screenshots / recordings
Test output
UI testing
6) Edge cases & considerations
require_versioning), the Versions tab and run dialog are hidden;compare/still works.expected_default_id; the loser gets a conflict.private_repository).draft_changed, and the restore does not bring the old draft back over that save.contract.json-only chat edit still reuses the resealed bundle (open, see below).generic-harness/left environment identity.Open items
contract.json-only chat edit still reuses the resealed bundle; the reseal does not refresh what a build derives from the contract (# TODO:indrafts._resealed).contract.jsonover the bundle's. Whether that can dropruntime.interfacefrom a sealed contract is not yet checked.dev, folding migrations 0098–0102 into one after dev's latest, and whether theauthor_hosted_harness_jobremoval moves to its own PR are pending and will be handled in a follow-up push.7) Pre-existing issues (NOT introduced by this PR)
None known.
8) Architectural / important decisions
--checkruns first.Backward compatibility
platform-vaultsecret refs keep resolving.secret-values/changed: it now requiresenvironment_idand returns 202 with the newagent_versioninstead of 201 withsecret_refs. In this repo, the create page now sendscredential_valueswith the create request, andchangeCredentialsinversions.jssendsenvironment_id.author_hosted_harness_jobactivity is removed; deploy workers with the API.9) End-to-end test on the local stack
main. The poller sees no change on the built commit, the Run dialog shows no warnings, and the baseline run completes.geocode_address→search_places). The environment and scenario rebuilds land, and every scenario key stays the same.draft_stale.rebuild_running. A run-chat save between the rebuild's checks and its landing is not refused; the landing then fails withdraft_changed(covered by tests only).03d7d899certifies and its bundle seeds 7.25.86ef4ef6certifies and its bundle seeds 7.50.world-ir.jsonstill holds 60. Runf428502fcertifies fromworld.sqliteand seeds 61.50.generic-harness/left environment identity: environment v13 → v14 with no world edit; the change summary counts two world files changed.59572bb5certifies the pair with--certify-onlyand completes 1/1 calls.eb3aba24): the cached bundle replays and the run completes.--checkreports 6 environments current, 0 errors; the operator form runs (current 6).Checklist
bin/testpasses locally (backend)yarn test:runpasses locally (frontend)yarn contracts:checkpasses if API surface changed