test: enforce the everyday-work qualification checkpoint by cryppadotta · Pull Request #17090 · paperclipai/paperclip · GitHub
Skip to content

test: enforce the everyday-work qualification checkpoint - #17090

Open
cryppadotta wants to merge 7 commits into
codex/dependable-question-journeyfrom
codex/dependable-qualification-checkpoint
Open

cryppadotta wants to merge 7 commits into
codex/dependable-question-journeyfrom
codex/dependable-qualification-checkpoint

Conversation

@cryppadotta

@cryppadotta cryppadotta commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thinking Path

  • Paperclip must deliver useful results through complete user journeys.
  • A green run badge or a later retry can hide an earlier failure.
  • Dependable everyday work needs a fixed, inspectable qualification checkpoint.
  • The existing Product E2E reports already retain attempts, costs, timings, and evidence.
  • This PR grades the entire retained pool before best-attempt selection.
  • It requires the agreed candidate repetitions and later staging proof for every core cell.
  • It provides exact daily selections for the existing operations owner without another scheduler.

Linked Issues or Issue Description

Refs #17089 and #17088. The daily operations integration is owned by #17051; this PR adds no competing scheduler or spending mechanism.

What happened?

Partial green development cells can be mistaken for complete everyday-work qualification. Best-attempt report selection can also conceal the failures that preceded a successful retry.

Expected behavior

Require six journeys on Codie and Claude through Slack and Paperclip. Each of the 24 cells needs three fresh candidate passes on a recorded revision and subsequent staging proof. Retain every failure. Development cells and missing observer evidence cannot count.

Steps to reproduce

Run the checkpoint tests, or generate a report with PAPERCLIP_DEPENDABLE_QUALIFICATION_MANIFEST pointing at a private qualification manifest. An incomplete checkpoint must fail the report command while preserving every attempted result.

What Changed

  • Define the six journeys and 24 required cells with independent output, state, rendering, delivery, and journey-specific assertions.
  • Validate source, serving revisions, model, settings, instructions, runner artifact, target, freshness, and distinct task/run identities.
  • Add a typed observer receipt to retained results. Reject unexpected fields, including inherited object keys, without echoing untrusted field names.
  • Grade the full report pool before choosing its displayed best attempts. Emit a checkpoint report and fail an incomplete checkpoint.
  • Return four explicit daily execution IDs, rotating all cells in six UTC days. Refuse selection while any required ID is absent from the catalog.
  • Preserve ordinary billing, timings, evidence scanning, cleanup, and report ownership.

Verification

  • Final Product E2E support: 3,368 passed, two existing skips; source prerequisite checks: 128 passed.
  • Product E2E typecheck passes.
  • Calibration requires 72 fresh candidate passes plus later staging proof, preserves failures after successful retries, and rejects wrong runtime, stale settings, changed source, missing output checks, reused tasks/runs, incomplete cleanup, or missing observer receipts.
  • The actual report command is tested with retained failed and passing attempts. It writes an incomplete checkpoint and exits with failure.
  • Five new regressions fail before the metadata and date corrections, then all 37 checkpoint checks pass.
  • Full core journey registration and staging/Slack observers remain incomplete. Existing development cells emit no qualifying receipt. No live qualification, reliability percentage, or enabled daily campaign is claimed.
  • Review found two report defects. Both reproduce through the actual report command: missing proof could count, and duplicate receipts aborted report output. The repair rejects absent, linked, or unsafe manifest-listed files and retains duplicate receipts as incomplete observations while writing the ordinary reports. All four report calibrations pass.
  • Exact-head CI and review are pending. Full repository checks will run in CI; targeted local checks are not presented as full repository verification.

Risks

The checkpoint depends on correctly attributed observer evidence from the remaining live journeys. Its schema and negative tests cannot establish that those journeys passed. Daily selection deliberately fails until every required exact ID exists. The existing operations manifest and journal must control scheduling and spending. Only an explicitly supplied private manifest enables this additional report gate.

Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose its exact deployment ID or context-window size. Those values are not inferred.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread tests/runner-e2e/dependable-qualification.ts
Comment thread tests/runner-e2e/dependable-qualification.ts Outdated
@ponytail-paperclip-review

ponytail-paperclip-review Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

cryppadotta and others added 2 commits October 21, 2026 08:24
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* codex/dependable-repository-conversion:
  fix: validate actual task roots and unfinished response waits
  test: qualify conversion prerequisites before runner selection
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Comment thread tests/runner-e2e/dependable-qualification.ts Outdated
cryppadotta and others added 2 commits October 21, 2026 08:40
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* codex/dependable-repository-conversion:
  test: create conversion-compatible Claude baseline settings
  fix: preserve plain task sources through shared workspace reuse
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@cryppadotta
cryppadotta changed the base branch from codex/dependable-repository-conversion to codex/dependable-question-journey October 21, 2026 14:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant