fix(evals): retain feedback after worker interruption by cryppadotta · Pull Request #17092 · paperclipai/paperclip · GitHub
Skip to content

fix(evals): retain feedback after worker interruption - #17092

Open
cryppadotta wants to merge 3 commits into
codex/eval-operations-schedulingfrom
codex/eval-operations-feedback-recovery
Open

cryppadotta wants to merge 3 commits into
codex/eval-operations-schedulingfrom
codex/eval-operations-feedback-recovery

Conversation

@cryppadotta

@cryppadotta cryppadotta commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thinking Path

  • Paperclip manages AI agents and their work.
  • Product E2E retains feedback before it removes a test company.
  • A killed test worker can skip that collection and its result file.
  • Removing the local database then loses the saved feedback.
  • The server supervisor can read it before stopping the isolated server.
  • This change retains that evidence without turning an interrupted test into a pass.

Linked Issues or Issue Description

Refs #17017, #17043, and #17051. This follow-up is stacked on #17051.

What Changed

  • Embed attempt identity in every atomic recovery checkpoint, so saved feedback remains readable if the final receipt write times out.
  • Give supervisor shutdown a shared five-second feedback recovery window for unfinished local operator fixtures.
  • Save recovery evidence separately. Keep prior worker checkpoints when a later read fails or omits previously captured records.
  • Bind recovered counts to campaign, execution, attempt and company identity. Include them in the failed synthetic result.
  • Retain the private recovery database when the worker result is missing. Report unconfirmed cleanup instead of deleting it.
  • Leave completed results unchanged. Keep active-run coverage partial and failed collection visible.

Verification

  • 46 initial focused recovery, feedback, cleanup and shutdown checks passed; 42 report/recovery checks also passed. The final timeout fix passed 32 focused checks and eval typecheck. The new regression preserves an earlier saved body after a later write exhausts the deadline.
  • A real API/PostgreSQL test passed: a separate process submitted a complaint and suggestion, received both acknowledgments, and was killed with SIGKILL. Recovery retained both exact bodies and their run/agent/task attribution. Both remained readable after database teardown. Coverage remained partial, and no model calls were made.
  • The first integration invocation could not start PostgreSQL because the restored checkout lacked the package's library symlinks. Running its declared hydration script repaired the dependency. The failed log is retained.
  • The recovery branch passed a full build and workspace typecheck before the final narrowly scoped timeout fix. The full host suite retained a Muse timeout after 22,955 passes; its focused reproduction passed. Remaining adapter/plugin projects passed after checkout recovery. The serialized lane retained three dependency-startup failures and one watchdog timeout. After the package symlink repair, database startup worked and the watchdog reproduction passed; one unrelated announcements authorization test retained a 15-second timeout. The initial timeout-regression invocation was blocked before startup by sandboxed Vite config writes; the permitted run passed. Full local acceptance is not claimed.
  • Current head 7150af7de6 passed CI. Greptile reviewed this head at 5/5. Its only finding is fixed and resolved.

Risks

  • Recovery is partial evidence. It cannot prove that an active run had no later feedback or complete tool invocation coverage.
  • An abrupt supervisor kill or unavailable API can prevent capture. The private database is retained, and cleanup remains unconfirmed.
  • The shutdown path has at most five seconds for collection before the existing server stop. Authenticated public-MCP fixtures do not use this operator path.
  • Retained failure state can consume local storage and contains private test data. It must be handled as recovery evidence.

Model Used

OpenAI GPT-6 via Codex, with reasoning, tool use and code execution. The exact runtime model ID and context-window size are not exposed to this session.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass (focused checks; full-suite limitations above)
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

cryppadotta and others added 2 commits October 21, 2026 08:09
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@ponytail-paperclip-review

ponytail-paperclip-review Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread tests/runner-e2e/feedback-recovery.ts Outdated
Co-Authored-By: Paperclip <noreply@paperclip.ing>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant