Repository navigation
Fix work-queue delivery recovery and formalize settlement guarantees - #69951
Conversation
Validate native PR transport metadata against captured delivery evidence. Bind inline termination to the authenticated native principal and reconcile released pending deliveries conservatively after their terminal deadline. Preserve private delivery diagnostics, expose worker outcomes, and require evidence-backed ESLint research and failure-preserving validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Specify authenticated cleanup, released original-Claim recovery, corroborated PR transport, conservative deadline expiry, and scoped diagnostics. Add a bounded TLA+ recovery model with conditional liveness, mutation controls, guarded witnesses, and exact-verdict runner coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
REQUEST_CHANGES
The released-delivery recovery change still drops at least one retained-claim path, so the new settlement guarantee is not actually preserved.
Blocking theme
actions/setup/js/work_queue_reconciler.cjs: released candidates now depend onwork.claim_id === member.claim_id, which excludes re-grantedkeep_openwork even though the original claim barrier is still pending. That can strand delivery recovery forever after capacity release.
Static review only; I did not run the repo code or tests.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 88 AIC · ⌖ 6.06 AIC · ⊞ 12.7K
Comment /review to run again
| dispatch.claims?.some(member => { | ||
| const claim = projection.claims.get(member.claim_id); | ||
| const work = projection.works.get(member.work_id); | ||
| return claim?.state === "completed" && work?.claim_id === member.claim_id && (claim.keep_open ? claim.barrier : work.barrier) === "pending"; |
There was a problem hiding this comment.
The new released-dispatch candidate filter still requires work.claim_id === member.claim_id, so a keep_open claim can fall out of maintenance as soon as the Work is re-granted and its original delivery barrier will stay pending forever.
💡 Why this blocks merge
This path is supposed to keep reconciling released dispatches until the original ordinary or retained barrier is settled, even after capacity is released. For retained claims, a later re-grant is exactly the case where work.claim_id changes while the original claim barrier is still the thing that must be closed. The new predicate rejects that state before maintenance ever calls finalizeWorkerResults, so the sweep can never publish Result or DeliveryFailure for the original completed claim.
A safer check is to gate ordinary completions on work.claim_id === member.claim_id, but for keep_open use the original claim barrier alone and let the finalizer re-check ownership before publishing.
There was a problem hiding this comment.
Fixed in e863acf: released retained-Claim candidates remain eligible based on the original pending Claim barrier after Work ownership changes. The finalizer retains its ownership recheck before publishing.
There was a problem hiding this comment.
Skills-Based Review ðŸ§
Applied /diagnosing-bugs and /tdd (via pr-triage, change_type: bug_fix) against the highest-impact files: finish_work_queue_claim.cjs, reconcile_work_queue_run.cjs, work_queue_reconciler.cjs, work_queue_native_creation_delivery.cjs, safe_output_handler_manager.cjs.
📋 Key Themes & Highlights
Positive Highlights
- � Textbook
/diagnosing-bugsdiscipline: each of the three described bugs (PR transport rejection, missing-principal cleanup, stuck released reservations) ships with a targeted regression test reproducing the exact failure mode before the fix (e.g.it.each([-1, 0])("allows missing-verifier expiry only at the exact delivery deadline...). - �
reconciliationCandidatesandfinalizeWorkerResultschanges are conservative by construction — expiry only fires at/after the exact deadline with authenticated terminal evidence, effects stayunknownrather than being inferred, and no Result/replay path was touched. This matches the PR's stated review boundaries and is backed by new TLA+ invariants (ResultAuthority,PublicationFence) with 18 checked configurations. - âœ� Negative-path test coverage for the transport-metadata fix is thorough:
it.eachover foreign repo, null fields, invalid base_commit, and an "ancestry claimed but not proven" case that correctly still rejects verification. - � Diagnostics change (
claim-delivery-errors.json) correctly preserves the original handler failure report rather than overwriting it, and the new Go compiler test assertswork_queue_outcomeis worker-only andclaim_resultsnever leaks as a raw job output.
Minor, Non-blocking Suggestions
- Two small
/codebase-designnits posted inline: (1) the newclaim-delivery-errors.jsonfilename is a duplicated string literal across.cjs/.go/test files — consider a shared constant; (2) the two near-duplicateverification_unavailablepending branches infinish_work_queue_claim.cjs(lines ~286, ~313) would benefit from a short comment distinguishing their guard conditions.
Scope Note
The diff is large (68 files, many .lock.yml/.tla/.cfg regenerated artifacts) but the actual logic changes are concentrated in the ~7 files reviewed above; those were read in full alongside their test deltas, so confidence in this review is high despite the overall diff size.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 83.4 AIC · ⌖ 15 AIC · ⊞ 10.1K
Comment /matt to run again
| { | ||
| type: "claim_delivery", | ||
| errorCode: result?.reason || "verification_unresolved", | ||
| error: settlement.delivery?.reason || settlement.error || "Independent delivery verification is unavailable", |
There was a problem hiding this comment.
[/codebase-design] The claim-delivery-errors.json filename is a hardcoded string literal here, duplicated again in the test file. Consider exporting a CLAIM_DELIVERY_ERRORS_FILE_NAME constant from safe_output_manifest.cjs alongside SAFE_OUTPUT_ERRORS_FILE_PATH, matching the existing convention for the sibling error-report path.
💡 Why this helps
Keeping the filename as a shared constant avoids silent drift if the artifact name changes later (it already needed a matching update in compiler_safe_outputs_steps.go and compiler_safe_outputs_job_test.go), and makes the relationship between the two error-report files explicit.
@copilot please address this.
There was a problem hiding this comment.
Fixed in e863acf: the manager and regression test now use the exported CLAIM_DELIVERY_ERRORS_FILE_NAME constant from safe_output_manifest.cjs.
| integer(work.completion_at, 0, Number.MAX_SAFE_INTEGER, "Completion timestamp"); | ||
| const remainingDeadline = policy.deadline_ms - Math.max(0, (options.now ?? verificationStarted) - work.completion_at); | ||
| if (typeof options.verifyEffects !== "function" && remainingDeadline > 0) { | ||
| results[member.handle] = { state: "pending", effects: "unknown", reason: "verification_unavailable" }; |
There was a problem hiding this comment.
[/codebase-design] The verification_unavailable pending-state branch is now duplicated (lines ~286 and ~313) with slightly different guard conditions (expireUnverified !== true vs remainingDeadline > 0). A short comment above each branch (or a small named helper) clarifying why both checks exist would help a future reader avoid conflating or collapsing them incorrectly.
ðŸTM¡ Suggestion
This is purely a readability nit — the logic itself (verified via the new it.each([-1, 0]) deadline-boundary test) looks correct.
@copilot please address this.
There was a problem hiding this comment.
Fixed in e863acf: comments now distinguish the no-verifier/expiry opt-in guard from the separate Completion-deadline guard.
|
@copilot run pr-finisher skill |
There was a problem hiding this comment.
🔵 Needs a closer look
Irreversible settlement spans authentication and concurrent ledger updates, with unresolved recovery and transport issues requiring human review.
4 open issues
What changed in this PR
Repairs gh-aw work-queue delivery recovery while separating capacity release from delivery settlement.
Changes:
- Recover released pending deliveries with authenticated, deadline-gated failure handling.
- Add scoped diagnostics, worker outcomes, and stronger ESLint Miner instructions.
- Document settlement guarantees and add recovery models and regression coverage.
| File | Description |
|---|---|
| specs/work-queue/verify_models_test.py | Check exact recovery-model verdicts. |
| specs/work-queue/RecoveryUnavailableWitness.cfg | Witness unavailable-evidence pending state. |
| specs/work-queue/RecoveryRetainedWitness.cfg | Witness retained-Claim recovery. |
| specs/work-queue/RecoveryReleasedWitness.cfg | Witness recovery after release. |
| specs/work-queue/RecoveryEnrichedWitness.cfg | Witness enriched PR verification. |
| specs/work-queue/README.md | Explain recovery models and limitations. |
| specs/work-queue/DeliveryRecoveryProgress.cfg | Check conditional settlement progress. |
| specs/work-queue/DeliveryRecovery.cfg | Check recovery safety. |
| specs/work-queue/ClaimDeliveryRecovery.tla | Model delivery recovery guarantees. |
| specs/work-queue/check.sh | Register recovery checks and controls. |
| specs/work-queue/BrokenRecoveryUnfair.cfg | Demonstrate fairness dependency. |
| specs/work-queue/BrokenRecoveryRetained.cfg | Detect invalid retained re-grants. |
| specs/work-queue/BrokenRecoveryReplay.cfg | Detect effect replay. |
| specs/work-queue/BrokenRecoveryReleased.cfg | Detect released-candidate starvation. |
| specs/work-queue/BrokenRecoveryPrincipal.cfg | Detect incorrect cleanup authority. |
| specs/work-queue/BrokenRecoveryMetadata.cfg | Detect rejected corroborated metadata. |
| specs/work-queue/BrokenRecoveryForeignMetadata.cfg | Detect accepted foreign metadata. |
| specs/work-queue/BrokenRecoveryEvidence.cfg | Detect invalid terminal evidence. |
| specs/work-queue/BrokenRecoveryEffects.cfg | Detect incorrect effect disposition. |
| specs/work-queue/BrokenRecoveryDeadline.cfg | Detect premature expiry. |
| specs/work-queue/BrokenRecoveryCAS.cfg | Detect stale publication. |
| specs/work-queue/BrokenRecoveryArtifact.cfg | Detect artifact-derived authority. |
| pkg/workflow/compiler_safe_outputs_steps.go | Include delivery diagnostic uploads. |
| pkg/workflow/compiler_safe_outputs_job.go | Expose worker outcome classification. |
| pkg/workflow/compiler_safe_outputs_job_test.go | Cover outcomes and artifact paths. |
| docs/src/content/docs/specs/work-queue-specification.md | Define settlement requirements and model boundaries. |
| docs/src/content/docs/reference/work-queue.md | Describe recovery and diagnostics. |
| actions/setup/js/work_queue_reconciler.cjs | Select released deliveries and reconcile settlement. |
| actions/setup/js/work_queue_native_creation_delivery.test.cjs | Cover enriched PR metadata verification. |
| actions/setup/js/work_queue_native_creation_delivery.cjs | Corroborate supported PR transport fields. |
| actions/setup/js/work_queue_lifecycle.test_helpers.cjs | Model authenticated AW credential principals. |
| actions/setup/js/safe_output_routing_parity.test.cjs | Cover separate delivery diagnostics. |
| actions/setup/js/safe_output_handler_manager.cjs | Write scoped delivery failure reports. |
| actions/setup/js/reconcile_work_queue_run.test.cjs | Cover authenticated cleanup and scheduled recovery. |
| actions/setup/js/reconcile_work_queue_run.cjs | Preserve cleanup identity and detect unresolved delivery. |
| actions/setup/js/finish_work_queue_claim.test.cjs | Cover missing-verifier deadline boundaries. |
| actions/setup/js/finish_work_queue_claim.cjs | Permit explicit missing-verifier expiry. |
| .github/workflows/repo-tree-map.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/eslint-refiner.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/eslint-monster.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/eslint-miner.md | Require research and checked validation. |
| .github/workflows/eslint-miner.lock.yml | Refresh prompt hash, outcome, and diagnostics. |
| .github/workflows/engine-conformance-pydantic-ai.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-pi.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-opencode.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-kiro.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-goose.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-gemini.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-deepseek-harness.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-cursor.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-crush.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-copilot.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-codex.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-claude.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-aider.lock.yml | Expose worker outcome. |
| .github/workflows/engine-conformance-agy.lock.yml | Expose worker outcome. |
| .github/workflows/deep-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-token-consumption-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-team-evolution-insights.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-secrets-analysis.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-repo-chronicle.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-regulatory.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-observability-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-issues-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-firewall-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-evals-report.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/daily-compiler-quality.lock.yml | Expose outcome and delivery diagnostics. |
| .github/workflows/artifacts-summary.lock.yml | Expose outcome and delivery diagnostics. |
Review effort: Max
| if ((message.head_repo !== undefined && message.head_repo !== receipt.headRepository) || (message.base_branch !== undefined && message.base_branch !== receipt.baseBranch)) return { verified: false }; | ||
| if (message.base_commit !== undefined) { | ||
| if (typeof message.base_commit !== "string" || !REVISION.test(message.base_commit)) return { verified: false }; | ||
| if (message.base_commit !== receipt.base && !(await readAncestry(github, receipt.repository, message.base_commit, receipt.base))) return { verified: false }; |
There was a problem hiding this comment.
Fixed in e863acf: the private receipt now captures the transport base branch/commit separately from the effective PR base. Regression cases cover direct overrides and resolved stack branches while retaining independent ancestry checks.
| if (dispatch.state !== "reserved" && typeof options.verifyEffects === "function") await require("./finish_work_queue_claim.cjs").finalizeWorkerResults({ ...options, assignment }); | ||
| results.push({ dispatch_id: dispatch.dispatch_id, ...(await reconcileDispatch({ ...options, assignment })) }); | ||
| const reservation = await reconcileDispatch({ ...options, assignment }); | ||
| const delivery = dispatch.state !== "reserved" && (dispatch.run || reservation.released) ? await require("./finish_work_queue_claim.cjs").finalizeWorkerResults({ ...options, assignment, expireUnverified: true }) : undefined; |
There was a problem hiding this comment.
Fixed in e863acf: reconciliation finalizes delivery only when a verifier or explicit expiry opt-in is present; maintenance now explicitly opts in. Tests cover omitted and false opt-in.
| const delivery = dispatch.state !== "reserved" && (dispatch.run || reservation.released) ? await require("./finish_work_queue_claim.cjs").finalizeWorkerResults({ ...options, assignment, expireUnverified: true }) : undefined; | ||
| results.push({ dispatch_id: dispatch.dispatch_id, ...reservation, ...(delivery ? { delivery } : {}) }); |
There was a problem hiding this comment.
Fixed in e863acf: reservation and delivery finalization now have separate error handling. A delivery exception preserves the reservation result and reports unresolved pending claims; the maintenance regression test covers this.
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Completed the PR-finisher pass and pushed |


Why
The ESLint worker investigation found that a worker could publish a useful PR and record Completion without reaching a verified Result. Enriched PR transport fields were rejected, inline cleanup used a missing logical-profile principal, and maintenance stopped selecting pending deliveries after their native reservations were released. This could leave Work and dependent DAG nodes stranded despite successful external effects.
Approach
DeliveryFailurewith effectsunknown. Never fabricate Result, infer no effects, or replay mutations; retain original-Claim and CAS fencing.claim-delivery-errors.jsondiagnostics. Expose only the fixedwork_queue_outcomeclassification as a worker job output. Regenerate the 30 affected worker locks.Validation
make build,make fmt, JavaScript typecheck, andmake lint-cjspassed. Existing lint warnings remain.make recompile: 344 workflows compiled. Current-compiler differential actionlint/zizmor/poutine/shellcheck validation added no findings; baseline findings remain, so this is not a clean-scan claim.PATH="/opt/homebrew/bin:$PATH" make agent-report-progresspassed for the runtime repair, including impacted Go tests and 247 selected JavaScript tests. The formal-only follow-up passed the same gate scoped to the runtime commit; unchanged Go/workflow tests were skipped.python3 -B specs/work-queue/verify_models_test.pypassed all 11 runner tests, andgo test ./pkg/workqueue -run '^TestDeliveryExhaustionDeadlineAndAttemptNativeParity$' -count=1passed all 16 shared deadline cases. SANY semantic analysis and shellcheck passed.Review boundaries
Capacity release is deliberately not delivery success. Expiry can close an unresolved barrier as unknown-effects failure even when an external PR exists; it does not make that Work safe to rerun automatically. The TLA+ liveness result is bounded and conditional on fair clock, maintenance and publication actions plus eventually stable exact-terminal observations. It does not prove GitHub API behavior, actual CAS/WeakMap implementation, arbitrary candidate-population fairness, or runtime refinement.
No live queue clearing, ledger mutation, workflow dispatch/cancellation, or effect replay was performed. The strengthened worker prompt is an instruction, not a formal guarantee of agent compliance.