Repository navigation
fix(orchestration): deliver worker-exit escalations to lightweight Run coordinators (STA-4604) - #15235
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review. 📝 WalkthroughWalkthroughWorker-exit escalation now records the settled dispatch result and resolves recipients by orchestration mode. Lightweight runs use their 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts (1)
8-12: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCondense the test rationale comments.
The test name and assertions state the tested behavior. Reduce these comments to brief statements of the legacy-versus-lightweight routing rule.
As per coding guidelines: “Comments must be concise, non-obvious, and brief—prefer one line; do not explain obvious behavior.”
Also applies to: 103-104
Source: Coding guidelines
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 627d3dc1-71e3-45df-a710-0bdf21d5e395
📥 Commits
Reviewing files that changed from the base of the PR and between 19ba83d and e8ea7d0696b46532133ab04d19e07de4d60dfaf5.
📒 Files selected for processing (2)
src/main/runtime/orca-runtime.tssrc/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
Included review availability: Your plan includes up to 10 reviews per rolling hour; 6 remain after this review.
Review round 1 — 3 independent
|
| Mutation | Caught by |
|---|---|
drop runId |
lightweight + isolation |
address bare handle instead of run:<id> |
new unbound-Run case |
delete notifyMessageArrived |
wake test |
| invert the legacy check | 7 tests |
| drop the missing-owner fallback | new orphaned-dispatch case |
| skip the escalation entirely | 7 tests |
| resolve a global "latest Run" | new isolation case |
The bare-handle mutation originally survived every test — trg_messages_route_coordinator_mail reroutes handle-addressed mail carrying a run_id, so the two addresses converge until the Run has no bound coordinator. That divergence is now pinned.
Also picked up from the routing review: the escalation payload now carries dispatchId. applyEscalationToDispatch rejects an escalation lacking an exact Dispatch binding, and a coordinator reading the message needs to know which Dispatch died.
Verification after hardening
9 tests in the new file; orchestration suite 51 files / 425 tests pass; src/main/runtime 4577 pass with only the pre-existing quarter-circle-title-send-authorization failures, which fail identically on clean main. check:code-quality:changed, typecheck, and oxfmt all clean.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts (1)
106-115: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the exact dispatch ID.
expect.any(String)accepts an escalation for a different dispatch. Returndispatch.idfromgradeWorkerExitand assert it inpayload.dispatchId.Proposed fix
dispatchStatus: string | undefined + dispatchId: string escalationsToCoordinatorHandle: number @@ dispatchStatus: db.getDispatchContextById(dispatch.id)?.status, + dispatchId: dispatch.id, @@ - dispatchId: expect.any(String), + dispatchId: graded.dispatchId,Also applies to: 141-162, 179-193
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6a6aad5c-fee5-4e21-9869-075b58543e87
📥 Commits
Reviewing files that changed from the base of the PR and between e8ea7d0696b46532133ab04d19e07de4d60dfaf5 and fe90ba1949bb4d7b3f349cd2f8dae1cda04c79ed.
📒 Files selected for processing (2)
src/main/runtime/orca-runtime.tssrc/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/main/runtime/orca-runtime.ts
Included review availability: Your plan includes up to 10 reviews per rolling hour; 4 remain after this review.
Review round 2 — both reviewers who requested changes now approveRouting reviewer — MAJOR withdrawn. After verifying the two facts, they confirmed They also traced the exact read path they'd originally thought was blocked: with no Test reviewer — APPROVE. Both prior MAJORs resolved; they independently confirmed the seven-mutation sweep is consistent with the new assertions. Remaining items are MINOR and explicitly non-blocking, recorded for honesty rather than fixed here:
Blast-radius reviewer: PASS with no findings from the first round. |
Review round 3 — three fresh
|
…n coordinators (STA-4604) failActiveDispatchOnExit failed the dispatch on a worker PTY exit but gated the "Agent exited unexpectedly" escalation on getActiveCoordinatorRun(), which reads the legacy coordinator_runs table. Lightweight Runs live in runs + run_coordinator_handles and never populate that table, so the gate never passed and worker death was silent. orchestration.run/runStop — the only writers of coordinator_runs — are retired at the RPC contract fence, so on current builds the escalation was unreachable for every Run, not just lightweight ones. Un-gating alone would not have delivered it: insertMessage defaults run_id to the legacy Run and addressed the bare coordinator handle, which getOrCreateRunDelivery (run_id = <run> AND to_handle = run:<id>) never reads and which trips legacy_read_only on handle reads. Route the escalation through the dispatch's own Run instead, addressing run:<id> the way the send path already does, and notify the mailbox so a coordinator blocked in `check --wait` wakes immediately rather than waiting for a poll that a dead pane can no longer prompt. Legacy Runs keep the legacy gate and handle addressing unchanged. Tests cover the lightweight delivery, legacy back-compat, the no-coordinator silence case, and the blocked-coordinator wake-up.
…review Review of #15235 found the suite could go green on real regressions. A mutation sweep over the fix now kills all seven mutations it was checked against: - The check --wait test was self-fulfilling: waitForMessage resolves off the notification alone and never reads the row, so a notify with no message behind it passed. It now asserts the message is actually in the Run mailbox. - The oracle graded counts only, so corrupted escalation content stayed green. It now pins from/to/run_id/type/priority/subject/delivery_contract/payload. - Nothing proved routing follows the dying worker's own Run. A second, newer Run is now live in the isolation test, which is what any "latest active run" lookup would wrongly pick — the exact shape of the bug being fixed. - Addressing the coordinator handle instead of run:<id> survived every test, because trg_messages_route_coordinator_mail reroutes handle-addressed mail with a run_id. The two only diverge once the Run has no bound coordinator, so an unbound-Run case now covers it. - Added supervised worker_dispatches, circuit-breaker, and missing-owning-Run fallback cases, none of which were exercised. Also carry dispatchId in the escalation payload: applyEscalationToDispatch rejects an escalation lacking an exact Dispatch binding, and a coordinator reading the message needs to know which Dispatch died.
…leanup onPtyExit walks every leaf on the dying pty synchronously and then prunes the pty records. Before STA-4604 the escalation was unreachable for lightweight Runs, so insertMessage never ran there and could not throw; now it runs on every worker death, putting a mailbox write on that path. A throw would abandon the remaining leaves and the pruning that closes out the exit. failDispatch is the authoritative transition and has already committed by then, so the mail is best-effort on top of it: catch and log instead of unwinding the exit.
ae11056 to
da22935
Compare
The escalation carried a subject line and a raw payload and nothing else, so the
banner a coordinator actually reads was a code and three ids. Learning which task
had died, or whether it was still retryable, meant looking the ids up by hand.
Add a body naming the task and stating the resulting state, taken from what
failDispatch settled to: a plain failure says the task is ready to dispatch again,
a tripped circuit breaker says it will not be retried automatically. Task naming
reuses buildOrchestrationTaskDisplayMetadata, so it matches the title shown
everywhere else rather than inventing a second derivation.
Before:
Subject: Agent exited unexpectedly (code 137)
[Payload: {"taskId":"task_c917b402bd49",...}]
After:
Subject: Agent exited unexpectedly (code 137)
Worker term_62dc5050 exited with code 137 while running task
"Migrate the billing schema to v2" (task_c917b402bd49). The task is ready to be
dispatched again.
[Payload: {"taskId":"task_c917b402bd49",...}]
Both bodies are pinned by tests; the grading oracle now compares the body so the
wording cannot rot silently.
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4bfc1721-4f1b-4d82-b891-bde5d1ff4b36
📥 Commits
Reviewing files that changed from the base of the PR and between ae11056b8f71c4e2303f494f3d2665ecf2c447e4 and 5527dff.
📒 Files selected for processing (2)
src/main/runtime/orca-runtime.tssrc/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
Included review availability: Your plan includes up to 10 reviews per rolling hour; 5 remain after this review.
The best-effort guard around worker-exit mail started at insertMessage, but resolveExitEscalationRecipient reads the database as well — getRun, and the legacy getActiveCoordinatorRun behind it. A failure there threw straight out of failActiveDispatchOnExit and produced the exact outcome the guard exists to prevent: onPtyExit abandons the remaining leaves and skips the pty record pruning that closes out the exit. Move the lookup inside the try. The catch logs run_id rather than the resolved recipient, since resolution is now one of the things that can fail. Caught by CodeRabbit on #15235.
Rebase onto #15235, which landed the delivery half of this: worker-exit escalations now reach lightweight Run coordinators instead of being gated on a legacy table nothing writes. That PR delivers the message; this one decides what the message says. Both edit the same block, so the resolution keeps every part of the delivery fix — run-aware addressing, the best-effort try/catch, the `notifyMessageArrived` wake, and the prose body — and swaps the exit code it described for the resolved cause. The escalation subject and body now name the cause rather than a code, since "exited with code 0" was the sentence that sent an operator chasing an OOM that never happened. The payload keeps `exitCode` for readers that already parse it and adds `exitCause` beside it. A deliberate close returns before the escalation, so closing a pane no longer pages a coordinator. Also sharpens an over-correction in the previous commit. Refusing to read *any* unreported bare code as a status was too broad: node-pty only ever pairs a signal with 0, and a wrapper spawn only ever returns 0, so a nonzero code really is the child's wait status and stays `exited(code)`. Only 0 is ambiguous, and only 0 now resolves to `unknown: cause_unreported`. That keeps a remote `exit 137` over the SSH relay informative while still refusing to call a possible SIGKILL a clean finish.
…n coordinators (STA-4604) (stablyai#15235)
…n coordinators (STA-4604) (stablyai#15235)
…n coordinators (STA-4604) (stablyai#15235)

Fixes STA-4604.
The bug
When a worker's PTY exits with an active dispatch,
failActiveDispatchOnExitfails the dispatch and is supposed to send the coordinator anAgent exited unexpectedlyescalation. The escalation was wrapped in a gate ongetActiveCoordinatorRun(), which reads the legacycoordinator_runstable. Lightweight Runs live inruns+run_coordinator_handlesand never populate it, so the gate never passed and worker death was silent.It is worse than the ticket reported:
orchestration.run/orchestration.runStop— the only production writers ofcoordinator_runs— are retired at the RPC contract fence (orchestration-contract-fence.ts,command_retired, no effects applied). On any current build nothing can populate that table, so the escalation was unreachable dead code for every Run, not just lightweight ones.Why un-gating alone wasn't enough
insertMessagedefaultsrun_idto the legacy Run and the old code addressed the bare coordinator handle. A lightweight coordinator consumes mail viagetOrCreateRunDelivery, which selectsrun_id = <run> AND to_handle = 'run:<id>'— so the message still would not have arrived, and a handle-addressed read tripslegacy_read_only.The fix
run_id. A modern Run getsto: run:<id>+runId, mirroring what theorchestration.sendpath already does for dispatch-derived routing. Legacy Runs keep the legacy gate and handle addressing, unchanged.notifyMessageArrivedafter inserting, the same convention every other message producer follows. Worker death is the one escalation nobody polls for — the dead pane can't nudge the coordinator, so a coordinator blocked incheck --waitpreviously sat until timeout.Tests
New
lightweight-run-worker-exit-escalation.test.tsdrives a real runtime + realOrchestrationDbthrough a synthetic worker PTY exit:run:<id>coordinator_runsrowcheck --waitnotifiedVerified red before / green after: reverting only the
orca-runtime.tshunk turns the two new-behavior tests red while both back-compat tests stay green.Verification
src/mainsuite: 2208 files / 23329 tests pass. The 4 failing files (pty-pane-claim-arbitration,port-scan-command-execution,quarter-circle-title-send-authorization,ssh-connection-sftp-namespace) fail identically on cleanmain— pre-existing and unrelated.check:code-quality:changed(incl. React Doctor) passes;typecheckpasses;oxfmtclean.