fix(orchestration): deliver worker-exit escalations to lightweight Run coordinators (STA-4604) by Jinwoo-H · Pull Request #15235 · stablyai/orca · GitHub
Skip to content

fix(orchestration): deliver worker-exit escalations to lightweight Run coordinators (STA-4604) - #15235

Merged
Jinwoo-H merged 5 commits into
mainfrom
readiness-p1-sta-4604-claude
Aug 18, 2026
Merged

Jinwoo-H merged 5 commits into
mainfrom
readiness-p1-sta-4604-claude

Conversation

@Jinwoo-H

@Jinwoo-H Jinwoo-H commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​479 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​479
Prod 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​79 $\color{#cf222e}{\Huge{\mathbf{−}}}$​10 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​69

Fixes STA-4604.

The bug

When a worker's PTY exits with an active dispatch, failActiveDispatchOnExit fails the dispatch and is supposed to send the coordinator an Agent exited unexpectedly escalation. The escalation was wrapped in a gate on getActiveCoordinatorRun(), which reads the legacy coordinator_runs table. Lightweight Runs live in runs + run_coordinator_handles and never populate it, so the gate never passed and worker death was silent.

It is worse than the ticket reported: orchestration.run / orchestration.runStop — the only production writers of coordinator_runs — are retired at the RPC contract fence (orchestration-contract-fence.ts, command_retired, no effects applied). On any current build nothing can populate that table, so the escalation was unreachable dead code for every Run, not just lightweight ones.

Why un-gating alone wasn't enough

insertMessage defaults run_id to the legacy Run and the old code addressed the bare coordinator handle. A lightweight coordinator consumes mail via getOrCreateRunDelivery, which selects run_id = <run> AND to_handle = 'run:<id>' — so the message still would not have arrived, and a handle-addressed read trips legacy_read_only.

The fix

  • Resolve the recipient from the dispatch's own run_id. A modern Run gets to: run:<id> + runId, mirroring what the orchestration.send path already does for dispatch-derived routing. Legacy Runs keep the legacy gate and handle addressing, unchanged.
  • Call notifyMessageArrived after inserting, the same convention every other message producer follows. Worker death is the one escalation nobody polls for — the dead pane can't nudge the coordinator, so a coordinator blocked in check --wait previously sat until timeout.

Tests

New lightweight-run-worker-exit-escalation.test.ts drives a real runtime + real OrchestrationDb through a synthetic worker PTY exit:

Case Asserts
lightweight Run escalation lands in the Run mailbox, addressed run:<id>
legacy Run + coordinator_runs row unchanged: delivered to the coordinator handle
legacy Run, no active coordinator unchanged: stays silent
coordinator blocked in check --wait wakes with notified

Verified red before / green after: reverting only the orca-runtime.ts hunk turns the two new-behavior tests red while both back-compat tests stay green.

Verification

  • Orchestration suite: 51 files / 420 tests pass.
  • Full src/main suite: 2208 files / 23329 tests pass. The 4 failing files (pty-pane-claim-arbitration, port-scan-command-execution, quarter-circle-title-send-authorization, ssh-connection-sftp-namespace) fail identically on clean main — pre-existing and unrelated.
  • check:code-quality:changed (incl. React Doctor) passes; typecheck passes; oxfmt clean.

@coderabbitai

coderabbitai Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 00c51982-f2ef-419d-9a73-8559d2954131

📥 Commits

Reviewing files that changed from the base of the PR and between 5527dff and 2d37625.

📒 Files selected for processing (2)
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

Worker-exit escalation now records the settled dispatch result and resolves recipients by orchestration mode. Lightweight runs use their run:<id> mailbox and include runId. Legacy runs use the active coordinator. The runtime skips insertion when no recipient exists, notifies recipients after insertion, and logs mailbox failures. Tests cover dispatch and worker settlement, delivery paths, coordinator wake-up, circuit-breaker behavior, orphaned dispatches, durable mailbox delivery, and insertion failures.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the worker-exit escalation routing fix for lightweight Run coordinators.
Description check ✅ Passed The description explains the bug, fix, rationale, linked issue, implementation details, tests, and verification results, but omits several template headings and checklist items.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts (1)

8-12: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Condense the test rationale comments.

The test name and assertions state the tested behavior. Reduce these comments to brief statements of the legacy-versus-lightweight routing rule.

As per coding guidelines: “Comments must be concise, non-obvious, and brief—prefer one line; do not explain obvious behavior.”

Also applies to: 103-104

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 627d3dc1-71e3-45df-a710-0bdf21d5e395

📥 Commits

Reviewing files that changed from the base of the PR and between 19ba83d and e8ea7d0696b46532133ab04d19e07de4d60dfaf5.

📒 Files selected for processing (2)
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 6 remain after this review.

@Jinwoo-H

Copy link
Copy Markdown
Contributor Author

Review round 1 — 3 independent gpt-5.6-luna (codex) reviewers

Blast-radius review: PASS, no findings. Verified reentrancy during onPtyExit, pointer coalescing vs. in-flight checks, multi-leaf/simultaneous exits, quit and mass-retirement lifecycle, and remote wire compatibility (no new field, opcode, or frame — existing escalation shape on existing mailbox columns).

Findings triaged

Routing review — MAJOR, adopted legacy Runs: not accepted. The claim is that adopt-legacy-run.ts creates the adopted Run with legacy = 0, so a retained legacy coordinator still on the legacy mail protocol misses a run:<id>-addressed escalation. Checked it and the premise doesn't carry:

  • getRunMailboxOwnerIdsForHandle (run-lookup.ts:34) deliberately maps that retained handle to the adopted Run, and orchestration-recipient-routing.ts:58 uses it to resolve handle-addressed mail to run:<adoptedRunId> — the routing layer's own canonical destination is exactly the address this PR uses.
  • It is not a regression either way: adoption sets every running row in coordinator_runs to failed, so pre-fix getActiveCoordinatorRun() returned nothing and that scenario got no message at all. The reviewer also concedes a current coordinator bound to the adopted Run works.

Test review — accepted, and acted on. The suite could have gone green on real regressions. Second commit fixes that; a mutation sweep over the fix now kills all seven mutations checked:

Mutation Caught by
drop runId lightweight + isolation
address bare handle instead of run:<id> new unbound-Run case
delete notifyMessageArrived wake test
invert the legacy check 7 tests
drop the missing-owner fallback new orphaned-dispatch case
skip the escalation entirely 7 tests
resolve a global "latest Run" new isolation case

The bare-handle mutation originally survived every test — trg_messages_route_coordinator_mail reroutes handle-addressed mail carrying a run_id, so the two addresses converge until the Run has no bound coordinator. That divergence is now pinned.

Also picked up from the routing review: the escalation payload now carries dispatchId. applyEscalationToDispatch rejects an escalation lacking an exact Dispatch binding, and a coordinator reading the message needs to know which Dispatch died.

Verification after hardening

9 tests in the new file; orchestration suite 51 files / 425 tests pass; src/main/runtime 4577 pass with only the pre-existing quarter-circle-title-send-authorization failures, which fail identically on clean main. check:code-quality:changed, typecheck, and oxfmt all clean.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts (1)

106-115: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the exact dispatch ID.

expect.any(String) accepts an escalation for a different dispatch. Return dispatch.id from gradeWorkerExit and assert it in payload.dispatchId.

Proposed fix
   dispatchStatus: string | undefined
+  dispatchId: string
   escalationsToCoordinatorHandle: number
@@
       dispatchStatus: db.getDispatchContextById(dispatch.id)?.status,
+      dispatchId: dispatch.id,
@@
-        dispatchId: expect.any(String),
+        dispatchId: graded.dispatchId,

Also applies to: 141-162, 179-193


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a6aad5c-fee5-4e21-9869-075b58543e87

📥 Commits

Reviewing files that changed from the base of the PR and between e8ea7d0696b46532133ab04d19e07de4d60dfaf5 and fe90ba1949bb4d7b3f349cd2f8dae1cda04c79ed.

📒 Files selected for processing (2)
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/main/runtime/orca-runtime.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 4 remain after this review.

@Jinwoo-H

Copy link
Copy Markdown
Contributor Author

Review round 2 — both reviewers who requested changes now approve

Routing reviewer — MAJOR withdrawn. After verifying the two facts, they confirmed run:<adoptedRunId> is the routing layer's own canonical destination for a retained legacy coordinator handle, and that adoption marks running coordinator_runs rows failed so the pre-fix path delivered nothing there. Their words: "This PR does not regress a previously delivered escalation in that scenario; it adds a current-delivery handoff for a legacy-compatibility consumer."

They also traced the exact read path they'd originally thought was blocked: with no --run, handleLegacyCheck surfaces a legacyCompatibility.currentDelivery hint containing orchestration check --run <adoptedRunId>; with that flag, resolveRunScope accepts the adopted Run and getOrCreateRunDelivery reads run:<adoptedRunId>. One honest residual they noted, which I'm recording rather than fixing here: a retained legacy coordinator already blocked in the no-arg legacy check waits on its terminal handle and won't be woken by this notification — it has to poll or follow the hint. That is a pre-existing property of the legacy compatibility read path, strictly better than the pre-fix silence, and out of scope for this PR.

Test reviewer — APPROVE. Both prior MAJORs resolved; they independently confirmed the seven-mutation sweep is consistent with the new assertions.

Remaining items are MINOR and explicitly non-blocking, recorded for honesty rather than fixed here:

  • settle() uses a real setTimeout(0), and each notifyMessageArrived schedules the runtime's unref'd 2s mail-pointer repoint timer, which can fire after the test closes its DB. The runtime swallows it; it's test hygiene, not correctness.
  • The isolation test has multiple live Runs but only one exiting worker — it pins owner selection, not concurrent-exit ordering.
  • Coverage stays local-Git-worktree-shaped; no folder-workspace or confirmed-SSH exit case. The resolver is independent of process.platform, worktree kind, and path resolution, and existing runtime tests cover the unconfirmed-SSH exit guard.

Blast-radius reviewer: PASS with no findings from the first round.

@Jinwoo-H

Copy link
Copy Markdown
Contributor Author

Review round 3 — three fresh gpt-5.6-luna (codex) reviewers, given the full round 1–2 record

Deliberately fresh agents rather than the round-1 reviewers, so approval wasn't just self-confirmation. Angles: adversarial (attack the prior rounds' conclusions), end-to-end coordinator behavior, and concurrency/lifecycle.

Fixed in ae11056b8f

Uncaught mailbox failure on the PTY-exit path — found independently by the concurrency and adversarial reviewers, and by me while they worked. onPtyExit walks every leaf synchronously and then prunes pty records. Before this PR the escalation was unreachable for lightweight Runs, so insertMessage never ran there and could not throw; now it runs on every worker death, putting a mailbox write on that path. A throw would abandon the remaining leaves — concretely, dispatch A on leaf A and dispatch B on leaf B sharing one pty: if A's insert throws, A is failed durably but B is never processed.

failDispatch is the authoritative transition and has already committed by then, so the mail is now best-effort on top of it: caught and logged rather than unwinding the exit. Covered by a test that asserts onPtyExit doesn't throw, the dispatch is still failed, and the failure is logged — and that test goes red if the guard is removed.

Not fixed here, recorded deliberately

MAJOR — restart/recovery deaths (adversarial reviewer). If the runtime is down or loses its exit subscription when a worker dies, no onPtyExit ever fires. A lightweight dispatch --inject has no worker_dispatches row, and startup recovery inner-joins that table, so it can't discover the dispatch; the stale-heartbeat path only logs. The dispatch can hold its task indefinitely with the coordinator silent.

This is real, but the reviewer states plainly it is not a regression from this PR — main has the same gap. Closing it means emitting the escalation from startup reconciliation and the stale-dispatch detector, which is a materially different change. This PR's contract is the exit event that does reach the runtime. Worth a follow-up ticket rather than scope creep here.

MINOR — adopted-legacy wake-up latency. Already recorded in round 2. A retained legacy coordinator blocked in no-arg check --wait waits on its bare handle and isn't woken by a run:<adoptedRunId> notification; it polls and gets a [CURRENT RUN MAIL WAITING] handoff. Still strictly better than the pre-fix silence. Both round-2 and round-3 reviewers independently agreed the original MAJOR was correctly withdrawn.

MINOR — formatMessagePointer hardcodes orca (behavior reviewer). Verified at formatter.ts:116. Real, but pre-existing and untouched by this PR — it affects every orchestration pointer, and WSL/dev runtimes need orca-ide/orca-dev. Separate fix.

Correction to my own earlier claim

The adversarial reviewer challenged my "7 pre-existing failures, identical on clean main," and they were right to. What I actually verified was narrow and sound: I stashed the change and ran those specific 4 files on clean main, getting the same 7 failures. What I should not have done is present a full-suite failure count as a stable baseline — it isn't. A later full src/main run on this branch produced 6 failing files, the two new ones being persistence-automations and terminal-history-tombstone-retry; both pass in isolation on this branch, i.e. load-sensitive flakes under three concurrent codex agents, matching the reviewer's own read that the extra failures were native-PTY/watcher noise.

The meaningful regression surface is clean: src/main/runtime + orchestration = 4583 passing, with only the known quarter-circle-title-send-authorization and pty-pane-claim-arbitration failures that I confirmed fail on clean main.

Current state

11 tests in the new file; orchestration suite 51 files / 426 tests pass; check:code-quality:changed, typecheck, oxfmt clean.

…n coordinators (STA-4604)

failActiveDispatchOnExit failed the dispatch on a worker PTY exit but gated the
"Agent exited unexpectedly" escalation on getActiveCoordinatorRun(), which reads
the legacy coordinator_runs table. Lightweight Runs live in runs +
run_coordinator_handles and never populate that table, so the gate never passed
and worker death was silent. orchestration.run/runStop — the only writers of
coordinator_runs — are retired at the RPC contract fence, so on current builds
the escalation was unreachable for every Run, not just lightweight ones.

Un-gating alone would not have delivered it: insertMessage defaults run_id to the
legacy Run and addressed the bare coordinator handle, which getOrCreateRunDelivery
(run_id = <run> AND to_handle = run:<id>) never reads and which trips
legacy_read_only on handle reads. Route the escalation through the dispatch's own
Run instead, addressing run:<id> the way the send path already does, and notify
the mailbox so a coordinator blocked in `check --wait` wakes immediately rather
than waiting for a poll that a dead pane can no longer prompt. Legacy Runs keep
the legacy gate and handle addressing unchanged.

Tests cover the lightweight delivery, legacy back-compat, the no-coordinator
silence case, and the blocked-coordinator wake-up.
…review

Review of #15235 found the suite could go green on real regressions. A mutation
sweep over the fix now kills all seven mutations it was checked against:

- The check --wait test was self-fulfilling: waitForMessage resolves off the
  notification alone and never reads the row, so a notify with no message behind
  it passed. It now asserts the message is actually in the Run mailbox.
- The oracle graded counts only, so corrupted escalation content stayed green. It
  now pins from/to/run_id/type/priority/subject/delivery_contract/payload.
- Nothing proved routing follows the dying worker's own Run. A second, newer Run
  is now live in the isolation test, which is what any "latest active run" lookup
  would wrongly pick — the exact shape of the bug being fixed.
- Addressing the coordinator handle instead of run:<id> survived every test,
  because trg_messages_route_coordinator_mail reroutes handle-addressed mail with
  a run_id. The two only diverge once the Run has no bound coordinator, so an
  unbound-Run case now covers it.
- Added supervised worker_dispatches, circuit-breaker, and missing-owning-Run
  fallback cases, none of which were exercised.

Also carry dispatchId in the escalation payload: applyEscalationToDispatch rejects
an escalation lacking an exact Dispatch binding, and a coordinator reading the
message needs to know which Dispatch died.
…leanup

onPtyExit walks every leaf on the dying pty synchronously and then prunes the pty
records. Before STA-4604 the escalation was unreachable for lightweight Runs, so
insertMessage never ran there and could not throw; now it runs on every worker
death, putting a mailbox write on that path. A throw would abandon the remaining
leaves and the pruning that closes out the exit.

failDispatch is the authoritative transition and has already committed by then, so
the mail is best-effort on top of it: catch and log instead of unwinding the exit.
The escalation carried a subject line and a raw payload and nothing else, so the
banner a coordinator actually reads was a code and three ids. Learning which task
had died, or whether it was still retryable, meant looking the ids up by hand.

Add a body naming the task and stating the resulting state, taken from what
failDispatch settled to: a plain failure says the task is ready to dispatch again,
a tripped circuit breaker says it will not be retried automatically. Task naming
reuses buildOrchestrationTaskDisplayMetadata, so it matches the title shown
everywhere else rather than inventing a second derivation.

Before:
  Subject: Agent exited unexpectedly (code 137)
  [Payload: {"taskId":"task_c917b402bd49",...}]

After:
  Subject: Agent exited unexpectedly (code 137)
  Worker term_62dc5050 exited with code 137 while running task
  "Migrate the billing schema to v2" (task_c917b402bd49). The task is ready to be
  dispatched again.
  [Payload: {"taskId":"task_c917b402bd49",...}]

Both bodies are pinned by tests; the grading oracle now compares the body so the
wording cannot rot silently.
@Jinwoo-H

Copy link
Copy Markdown
Contributor Author

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4bfc1721-4f1b-4d82-b891-bde5d1ff4b36

📥 Commits

Reviewing files that changed from the base of the PR and between ae11056b8f71c4e2303f494f3d2665ecf2c447e4 and 5527dff.

📒 Files selected for processing (2)
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 5 remain after this review.

Comment thread src/main/runtime/orca-runtime.ts
The best-effort guard around worker-exit mail started at insertMessage, but
resolveExitEscalationRecipient reads the database as well — getRun, and the legacy
getActiveCoordinatorRun behind it. A failure there threw straight out of
failActiveDispatchOnExit and produced the exact outcome the guard exists to
prevent: onPtyExit abandons the remaining leaves and skips the pty record pruning
that closes out the exit.

Move the lookup inside the try. The catch logs run_id rather than the resolved
recipient, since resolution is now one of the things that can fail.

Caught by CodeRabbit on #15235.
@Jinwoo-H
Jinwoo-H merged commit a27527a into main Aug 18, 2026
46 checks passed
Jinwoo-H added a commit that referenced this pull request Aug 18, 2026
Rebase onto #15235, which landed the delivery half of this: worker-exit
escalations now reach lightweight Run coordinators instead of being gated on a
legacy table nothing writes. That PR delivers the message; this one decides
what the message says. Both edit the same block, so the resolution keeps every
part of the delivery fix — run-aware addressing, the best-effort try/catch, the
`notifyMessageArrived` wake, and the prose body — and swaps the exit code it
described for the resolved cause.

The escalation subject and body now name the cause rather than a code, since
"exited with code 0" was the sentence that sent an operator chasing an OOM that
never happened. The payload keeps `exitCode` for readers that already parse it
and adds `exitCause` beside it. A deliberate close returns before the
escalation, so closing a pane no longer pages a coordinator.

Also sharpens an over-correction in the previous commit. Refusing to read *any*
unreported bare code as a status was too broad: node-pty only ever pairs a
signal with 0, and a wrapper spawn only ever returns 0, so a nonzero code
really is the child's wait status and stays `exited(code)`. Only 0 is
ambiguous, and only 0 now resolves to `unknown: cause_unreported`. That keeps
a remote `exit 137` over the SSH relay informative while still refusing to call
a possible SIGKILL a clean finish.
paidaxingyo666 pushed a commit to paidaxingyo666/Manta that referenced this pull request Aug 21, 2026
dallascrilley pushed a commit to dallascrilley/orca that referenced this pull request Aug 27, 2026
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant