fix(native): confirm remote runner exits during process checks by hsm-lv · Pull Request #17073 · paperclipai/paperclip · GitHub
Skip to content

fix(native): confirm remote runner exits during process checks - #17073

Open
hsm-lv wants to merge 1 commit into
paperclipai:masterfrom
hsm-lv:codex/fix-runner-exit-inspection
Open

hsm-lv wants to merge 1 commit into
paperclipai:masterfrom
hsm-lv:codex/fix-runner-exit-inspection

Conversation

@hsm-lv

@hsm-lv hsm-lv commented Oct 21, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work.
  • The control plane checks remote Runner processes before it releases ownership or starts recovery.
  • A process can exit between its command-line check and its start-generation read.
  • The old check reports missing process metadata as unavailable, even when the process has exited.
  • A zombie can also pass a signal-zero check after it has stopped work.
  • This pull request confirms these exits with a fresh ESRCH result or a matching zombie generation.
  • The active monitor can then finish the Run without treating a read failure as exit proof.

Linked Issues or Issue Description

Refs: #16835. That change requires confirmed exit before remote replacement. This change covers normal exits during those checks. I searched for related exit, fingerprint, zombie and ESRCH fixes. I found no equivalent open or merged change.

What happened?

A remote Runner can exit after the first liveness check. Its command line or process stat can disappear before the next read. The old check reports process_identity_changed, fingerprint_unavailable, or a generic probe failure. The active monitor cannot confirm exit. It can retain execution ownership after the Runner has returned its result. A matching zombie also has an empty command line and is rejected as an identity change.

Expected behavior

Confirm exit when a fresh signal-zero check returns ESRCH. Also confirm exit for a zombie whose boot identity and start ticks match the saved generation. Keep read failures and permission failures uncertain. Reject changed ownership markers and reused PIDs before any signal.

Steps to reproduce

  1. Start an owned child process with a saved marker and start generation.
  2. Let the command-line check succeed.
  3. Exit and reap the child before the separate fingerprint read.
  4. Observe the monitor result. Before the fix, it reports unavailable metadata instead of confirmed exit.

The regression tests also retire the child during the command-line read and supply a matching zombie stat fixture.

Paperclip version or commit

The patch is based on 8b873e01121a80e563a8a02bf6a3219783de5476.

Deployment mode

Remote Runner process supervision. The Linux probes were tested in a disposable Linux/amd64 container. The fault tests need no model provider or sandbox account.

What Changed

  • Share the process-generation check between active supervision and adopted-runner probes.
  • After ENOENT, require a fresh ESRCH result before confirming exit. Accept a zombie only when its generation matches.
  • Recheck liveness after a failed or mismatched command-line read. Use the system error code so EPERM cannot become exit proof.
  • Add tests for real child-process exit races, matching and reused zombie generations, unreadable metadata, changed markers and permission failures.
  • Document the exit evidence and recovery limits in the development guide.

Verification

  • Red: six selected regression cases fail against the unchanged baseline. The old code returns 45 for the initial permission failure, 1 for failed command-line reads, and 44 when the child exits during the command-line check.
  • Green: pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts passes all 798 tests. This includes the unchanged supervision and adoption cases.
  • pnpm --filter @paperclipai/server exec tsc --noEmit passes.
  • pnpm --filter @paperclipai/server exec tsc passes.
  • Five real Linux probe cases pass: live matching generation, EPERM under another user, changed generation, exit before the fingerprint read, and an already reaped PID. The expected exit codes are 0, 46, 47, 45 and 45.
  • The new liveness check starts one extra Node process inside the existing remote request. It uses the existing probe deadline. Five warm live probes had a median of 540 ms in an emulated Linux/amd64 container. This measurement is a local smoke check, not a production latency estimate. It adds no synchronous process call to the server event loop.
  • Standards and Spec reviews found no remaining confirmed issue. git diff --check passes.
  • Full pnpm -r typecheck and pnpm build stop because this machine has no Cargo. The server package's normal typecheck and build wrappers also stop when their vendor preparation calls that Rust build. The direct server TypeScript checks above pass after dependency builds. No full build or full-suite pass is claimed. The full test suite was not run because its Runner integration cases require the missing Rust binary.

Risks

  • Exit proof controls ownership. A false exit could permit concurrent work. The active monitor still requires the explicit process-not-running receipt. EPERM, unreadable stat data and changed generations cannot produce that receipt.
  • This patch keeps the adopted-runner isAlive() boolean interface. It does not add propagation of an unknown state through its existing callers.
  • Historical ownership holds still require separate recovery with explicit evidence. This patch does not release them automatically or replay completed work.
  • Each live probe starts one extra Node process. The remote runtime already requires Node. No new dependency, database migration, public API or configuration is required.

For core feature work, check ROADMAP.md first and discuss it in #dev before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See CONTRIBUTING.md.

Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code edits and test execution. The session does not expose the exact serving model ID or context-window size.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Supervision now requires Node

If Node is absent from the remote PATH, this unconditional call makes the first process check exit with 127. The monitor then rejects with probe_failed, even while the runner is still working.

The existing launch path allows this setup: REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT leaves the fingerprint empty when Node fails, and the normal launcher does not require a fingerprint. Previously, that path used shell commands alone for monitoring. Preserve it, or ensure Node is available before starting the runner. Add a regression test with Node absent from the remote PATH.

Prompt To Fix With AI
This is a comment left during a code review.
Path: server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts
Line: 109

Comment:
**Supervision now requires Node**

If Node is absent from the remote `PATH`, this unconditional call makes the first process check exit with 127. The monitor then rejects with `probe_failed`, even while the runner is still working.

The existing launch path allows this setup: `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` leaves the fingerprint empty when Node fails, and the normal launcher does not require a fingerprint. Previously, that path used shell commands alone for monitoring. Preserve it, or ensure Node is available before starting the runner. Add a regression test with Node absent from the remote `PATH`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant