What

gstack-eval-plan cli-run: a gstack-owned driver for subscription (CLI-metered) evals that stops at the first quota error. /eval-plan now routes subscription runs through it instead of telling agents to bring their own loop.

Why

An agent-written eval driver ran 1,164 headless claude -p calls in about 4 hours on one Claude subscription seat. It queued every job in a thread pool up front, used --output-format text, ignored exit codes and stderr, and retried unparseable output twice. When the five-hour window ran out, 389 calls were rejected in 4.5 minutes and the loop kept firing. Each rejected call left a transcript record with error: "rate_limit", apiErrorStatus: 429 and quotaLimits.status: "rejected", while rateLimitType was often null.

How

• Bounded pool. Exactly --workers (1–8, default 2) pull-based slots. A job starts only when a slot is free.
• Quota stop. Structured signals on stdout or stderr (error: rate_limit, 429, a rejected quotaLimits status, a rejected stream rate_limit_event), the CLI's limit message on a failed call, or exit 75 all end the run. No new job starts, unstarted jobs are recorded cancelled, and the run exits 75 with EVAL_QUOTA_STOP. A quota is never retried; --retries covers other errors only. Detection never relies on rateLimitType.
• No API fallback. The API billing variables are removed before the helper, the preflight or any child runs.
• Resume. Per-job JSONL records (0700 directory, 0600 files). A rerun skips ok jobs whose prompt and args hash is unchanged.
• Batch helper gate for full runs. --scope full re-runs itself under claude-batch run --repo <repo> --class standard|fable -- …. That external helper pins one seat and trips a durable circuit on the first quota error (GSTACK_CLAUDE_BATCH overrides the command). Inside the batch, the driver checks the marker env (MERIT_CLAUDE_BATCH_ID, overridable with GSTACK_CLAUDE_BATCH_MARKER), then runs <helper> preflight before the first call.
  • A missing helper is refused with EVAL_BATCH_HELPER_MISSING (exit 3), and nothing falls back to unwrapped calls.
  • A failed preflight, or a helper that never sets the marker, is refused with EVAL_BATCH_PREFLIGHT_FAILED.
  • --scope smoke (at most 10 jobs, 2 workers) needs no helper.
• API-funded eval routes are unchanged.
• Result codes. The three new codes live in lib/result-codes-eval-run.ts, spread into RESULT_CODES the same way as result-codes-release.ts. This keeps lib/result-codes.ts under the module-size limit. Each code has a docs/troubleshooting.md anchor.

Tests

test/eval-cli-run.test.ts (20 cases) drives the real bin against a fake claude and a fake claude-batch on a minimal PATH. No real model call and no network. The fake claude keeps its own in-flight counter under a lock, so concurrency is observed outside the runner. Coverage:

• the classifier on the real rejected-call record shape, null rateLimitType, stderr-only records, stream rate_limit_event, and answers that merely mention "rate limit"
• a quota stop that is never retried, with nothing starting after it
• exit 75 treated as fatal
• the in-flight bound
• the billing scrub at all three process layers
• resume, including a changed-args rerun
• file modes
• the helper gate: missing helper, failed preflight, wrap-then-preflight order, and a helper that never sets the marker
• smoke limits
• template structure

Mutation checks:

• removing the quota stop fails 5 tests
• retrying a quota fails 2
• removing the billing scrub fails 1
• passing the unscrubbed env to the helper fails 1
• the pre-review classifier and resume fail the 2 new review cases

Checks run

• bun test test/eval-cli-run.test.ts: 20 pass, 0 fail.
• Focused set (placeholders, troubleshooting anchors, skill validation, gen-skill-docs, pricing/eval-plan): 1207 pass, 0 fail.
• bun run typecheck: clean.
• bun run typecheck:test: no new diagnostics.
• bun run gen:skill-docs regenerated eval-plan/SKILL.md and the agents digest.
• The local bun run test did not pass on this machine: 434 failures on this branch, and 634 on base 952f95a. The failures are in PTY, browser, Docker, pregate and install tests. The files this change plausibly touches fail identically on HEAD and base when run alone (289 pass / 24 fail both). CI is the acceptance run.
• An outside review (GPT-5.6 Sol, xhigh) found two P1s and three P2s. All are fixed in 684e56a with regression tests.

Follow-ups

• ask: Once the external claude-batch helper is installed, run one --scope smoke and one tiny --scope full against it to confirm the live contract (marker, preflight, exit 75 propagation).

session: gstack-contrib-eval-quota-stop