test: verify repository delivery after real runner conversion by cryppadotta · Pull Request #17089 · paperclipai/paperclip · GitHub
Skip to content

test: verify repository delivery after real runner conversion - #17089

Open
cryppadotta wants to merge 8 commits into
codex/plain-task-repository-admissionfrom
codex/dependable-repository-conversion
Open

cryppadotta wants to merge 8 commits into
codex/plain-task-repository-admissionfrom
codex/dependable-repository-conversion

Conversation

@cryppadotta

@cryppadotta cryppadotta commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thinking Path

  • Paperclip manages agents that perform useful work.
  • Repository setup and Runner selection must preserve that work context.
  • A successful provider turn cannot prove configuration retention or usable delivery.
  • The repaired setup chain needs independent product checks with the recorded models.
  • This PR adds explicit repository and conversion journeys to the existing Product E2E harness.
  • The observer checks source bytes, Git identity, saved configuration, and downloaded output.
  • Development and staging qualification remain separate.

Linked Issues or Issue Description

Refs #17088 and #17047. This PR depends on the complete repository setup repair in #17088. No competing conversion journey was found in the open PR search.

What happened?

Existing checks do not prove that a real legacy agent retains its configured context when switched to Paperclip Runner and then delivers verified repository work. The projectless repair check also uses different default models from the recorded staging models.

Expected behavior

A configured repository task delivers correct, accessible output. A real legacy-to-native conversion preserves configuration, instructions, environment, and equivalent permission policy before doing subsequent work. A projectless task continues through setup without supervision.

Steps to reproduce

List the configured-repository, legacy-conversion, and automatic-repository-configured-models suites. Run one exact ID from the README. These are explicit paid development checks, excluded from the default selection.

What Changed

  • Add four configured-repository cells and four real legacy-conversion cells for Codex gpt-6.1-sol and Claude claude-opus-5-5 on local and Daytona environments.
  • Seed a fresh reference only in an owned disposable Git checkout. Compare the delivered JSON with its original bytes, hash, and commit.
  • Switch the same real legacy agent through the public Runner selection API. Compare allowlisted configuration and instruction digests before and after conversion.
  • Require one committed native run with the configured provider, model, and environment, a visible registered download after reload, Done, and no pending question.
  • Add four projectless replay cells with the recorded staging models and the unchanged strict continuation oracle.
  • Use production hire instructions and existing sandbox profiles, budgets, report pipeline, and cleanup. Record API-key and per-turn lifecycle differences explicitly.

Verification

  • Product E2E support after integrating the setup repair: 3,353 passed, two existing skips; source prerequisites: 128 passed.
  • Product E2E typecheck passes. Discovery lists all twelve new explicit cells.
  • Independent oracle calibration rejects wrong configuration, instructions, provider, model, source bytes, hash, commit, downloaded output, delivery, and pending questions.
  • The first full support run exposed a catalog count mismatch and an unprovisioned new profile ID. Both are fixed; the successful run uses the existing trusted sandbox profile IDs without changing workflow authority.
  • Review found fixture prerequisite defects: optional CLI-only Codex arguments and platform qualification after Runner selection. Both are corrected; the disposable Claude baseline also excludes its unsupported optional CLI turn limit. Seven new setup checks cover probe ordering, absent qualification, immutable images, local conversion, and uncertain probe accounting. The final focused cohort passes 86 checks and E2E typecheck passes.
  • The first qualification-regression run also had an incorrect test assumption about the legacy timeout; the corrected red reaches the unsupported extraArgs setting. No production conversion restriction was relaxed.
  • A retained four-cell development campaign delivered independently verified ordinary Codex output and real Codex/Claude conversion results with retained configuration. All three attempts nevertheless failed the required evidence contract because api-state.json was missing. The flow now saves allowlisted task/run lifecycle fields; it does not export prompts, comments, interactions, configuration contents, or provider outputs. The projectless cell delivered valid bytes but used a monitor fallback and failed the unchanged continuation oracle; its driver correction belongs to fix: repair the complete projectless repository setup chain #17088. All four attempts remain failures; fresh exact-candidate acceptance is pending. These API-key/per-turn checks do not establish subscription authentication, warm reuse, Slack delivery, or the complete milestone.
  • Full repository checks will run in CI. No full repository green claim is made from the targeted local checks.

Risks

The real provider may expose additional conversion or setup defects. Failures remain evidence and cannot be replaced by catalog registration or a later retry. The conversion uses the production API and does not inject a native result. Only synthetic data is seeded in a resolved fixture-owned directory. Configuration evidence contains digests, not raw environment values. This PR changes evaluation code, not production behavior.

Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose its exact deployment ID or context-window size. Those values are not inferred.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

cryppadotta and others added 2 commits October 21, 2026 07:50
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@ponytail-paperclip-review

ponytail-paperclip-review Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread tests/runner-e2e/repository-verification.ts Outdated
Comment thread tests/runner-e2e/repository-verification-flow.ts Outdated
cryppadotta and others added 2 commits October 21, 2026 08:06
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* codex/plain-task-repository-admission:
  fix: validate actual task roots and unfinished response waits

# Conflicts:
#	doc/plans/2026-10-20-dependable-everyday-work.md
Comment thread tests/runner-e2e/repository-verification.ts Outdated
cryppadotta and others added 2 commits October 21, 2026 08:38
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* codex/plain-task-repository-admission:
  fix: preserve plain task sources through shared workspace reuse
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cryppadotta and others added 2 commits October 21, 2026 09:13
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant