fix: stop idle execution-control recovery polling by cryppadotta · Pull Request #17094 · paperclipai/paperclip · GitHub
Skip to content

fix: stop idle execution-control recovery polling - #17094

Open
cryppadotta wants to merge 7 commits into
masterfrom
codex/execution-control-idle
Open

cryppadotta wants to merge 7 commits into
masterfrom
codex/execution-control-idle

Conversation

@cryppadotta

@cryppadotta cryppadotta commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thinking Path

Paperclip instances need to become quiet when they have no work, including releasing their idle database connections. Execution control still queried six recovery queues every 15 seconds after the MCP/Dot polling removal. These duties already have durable rows: most become actionable after a committed transition, while finalization and login cleanup have real deadlines. This PR routes those obligations through the existing transaction signals and delivery coordinator. Empty workers stop querying; unresolved work retains recovery.

Linked Issues or Issue Description

What happened?
On master c6ee96a3df, the execution-control interval repeatedly queries for abandoned finalization, safe replacements, continuation delivery, status delivery, automatic disposition, and expired local logins. This happens with empty queues and retained completed history while the server is active.

What did you expect to happen?
Inspect on startup, after relevant committed writes, or at a real deadline. Retry only while a concrete obligation or failed inspection remains unresolved.

Steps to reproduce
Start an otherwise empty server, let startup finish, and observe execution-control SQL continuing every 15 seconds. The new real-Postgres regression runs all six sweep implementations and checks that advancing beyond multiple former intervals produces no additional worker database calls.

Related work: #17036 and #15625. Searched for overlapping execution-control polling PRs; this extends the existing sleep work and does not add a new execution policy.

What Changed

  • Replace the unconditional interval with six independent coordinator workers. Slow cleanup cannot block unrelated delivery.
  • Notify through the outer transaction for run transitions, recovery-action writes, reconciliation, control-deadline creation/cancellation, and local login creation.
  • Restore deadlines and retry obligations from existing tables at startup. Reuse the owning replacement/disposition eligibility queries.
  • Bound each worker to the former 15-second scan budget even under repeated tenant writes; coalesce requested work without arming an empty worker. Retain the existing retry budget for pending batches, unresolved cleanup/delivery, or failed inspection. Retain wake generations across drain, concurrent writes, and failures.
  • Document the producer contract and the boundary between worker quiet and Cloud sleep authorization. No schema migration or new queue table.

Verification

  • Workspace pnpm -r typecheck: passed.
  • pnpm build: passed.
  • 543 tests passed across nine affected suites in one final targeted run. This included 14 execution-control worker tests plus native replacement, legacy recovery, coordinator, startup, recovery-action, explicit-continuation, and failed-workspace regression coverage.
  • New tests exercise all six real sweep implementations against PostgreSQL, outer commit/rollback, empty/history quiet, exact deadlines, earlier deadlines/cancellation, durable status retry/restart, continuation delivery, drain/resume, and independent worker shutdown.
  • Initial expanded testing exposed two fixture gaps: a recovery-action mock lacked the newly required transaction boundary, and a continuation fixture lacked the context that the production guarded update requires. Both fixtures were corrected; the final targeted run above includes both regressions.
  • After the shutdown and scan-budget refinements, 57 worker/startup tests passed, including 18 worker tests. A real Postgres regression sends 20 run transitions with a second company’s completed history present and verifies no extra worker SQL before the coalesced pass, then one delivery and a return to quiet. A separate 100-notification burst test verifies the same bound. The final server typecheck also passed.
  • The full local pnpm test:run attempt found four native-session integrity-test assertions that counted every post-provider transaction as recovery. Deadline publication now owns a separate transaction. Fault injection now targets the actual terminal-failure write; all 768 native-session tests passed after that correction, preserving protocol failure and no-provider-retry assertions. The original full-suite attempt was stopped after the correction to avoid duplicating final-head CI; it is not counted as a clean full run.
  • CI also identified a teardown fault-injection fixture still spying on db.update after the deadline write moved into a transaction. The fixture now intercepts the transactional write and preserves assertions for stopped renewal, released controls, and terminal status. Its focused regression, server typecheck, and all 407 heartbeat recovery tests passed.
  • Greptile rated 630c74c24a 5/5, no actionable issues. Security review identified tenant-write amplification from immediate global sweeps; the final refinement implements its bounded-coalescing option at the previous 15-second per-queue budget. The security finding is resolved and its re-scan passed. Server suites, typechecks, and build passed on that revision. The mocked Slack setup browser test failed twice at its success-heading assertion after reload. Its test now waits for the restored “Send a message” screen before simulating the webhook, explicitly checking resume readiness; the timeout and completion assertions are unchanged. Before the fix, eight isolated local repetitions and the full 25-test shard passed, confirming the failure was intermittent. With the new readiness assertion, five repetitions passed. Final-head CI and review are running.

Risks

The principal risk is an uninstrumented producer. Coverage includes the supported live server and shared createDb owner scope; independent SQL writers or separate replicas must participate in the notification contract. Durable rows remain authoritative and startup reconstructs obligations. Existing recovery authority/evidence, atomic guards, and retries are preserved.

This proves quiet for these workers, not provider suspension or fleet savings. Existing idle-safety blockers remain; no execution/login deadlines are newly exported to Cloud's external waker. Opted-in writes incur the existing transaction-ID notification cost.

Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution, and test tools. A more specific serving model identifier and context-window size are not exposed in this session. No subagents implemented this change.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

cryppadotta and others added 2 commits October 21, 2026 08:11
Drive execution-control obligations from committed writes and persisted deadlines while retaining scoped retries and restart recovery.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@ponytail-paperclip-review

ponytail-paperclip-review Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Co-Authored-By: Paperclip <noreply@paperclip.ing>

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

Comment thread server/src/services/heartbeat/run-lifecycle.ts
cryppadotta and others added 4 commits October 21, 2026 08:26
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant