decider: bound pending requests and enforce HTTP deadlines by Levius-Fubuki · Pull Request #149 · InferMatrixLab/system1-omni · GitHub
Skip to content

decider: bound pending requests and enforce HTTP deadlines - #149

Merged
hsliuustc0106 merged 17 commits into
InferMatrixLab:mainfrom
Levius-Fubuki:codex/decider-admission-deadline
Oct 24, 2026
Merged

hsliuustc0106 merged 17 commits into
InferMatrixLab:mainfrom
Levius-Fubuki:codex/decider-admission-deadline

Conversation

@Levius-Fubuki

@Levius-Fubuki Levius-Fubuki commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Refresh after #113 merged — 2026-10-24

Head 3476e5cb1f3182968fd82532ea0d7b9b28e7c5e2 normally merges latest main 3b3e9f434d89deb7e29def091d5e0ae52644f648 after #113 landed. This history-only merge has exactly the same source tree as the previously validated head 1718865f (6ed5ee3c71a5decc8058155de5468cbe430eac7e); no source or test files changed. The effective PR now contains only the admission/deadline increment above merged #113.

Fresh checks: formatting, Runtime/Decider tests (72 passed, 0 failed, 9 ignored), strict MkDocs and whitespace/merge checks passed. The preceding full workspace/release results below apply to the identical source tree; all preceding hosted jobs, including CUDA compilation, succeeded at 1718865f. New-head hosted checks are separate and rerun by GitHub. No new GPU execution or PR merge was performed.


Conflict resolution against main — 2026-10-24

Head 1718865f1c7297c125b4c783fef99bf4142bcea4 normally merges main 67b867b04b23acfe735c381559f7b2510f1e784e, retaining the original PR history. The only manual conflict resolution is the Qwen test manifest: Graph, prefix, constructor/stream-ownership, image-decoding and device-feature tests are all registered once. The merged Qwen source and tests match the refreshed #113; admission/deadline integration remains in this PR. ABI remains 9.

Current-head local macOS checks passed: formatting; strict workspace/all-target Clippy; workspace tests (348 passed, 0 failed, 65 ignored); workspace release build; 22 Decider Python tests; CUDA text contract (0 errors, existing Laya ABI warning); strict MkDocs; and whitespace/merge checks. Device/checkpoint cases were skipped; no new CUDA compilation, GPU parity or performance qualification was performed. Earlier A800 results remain pinned to their original source. Hosted checks for this new head are reported by GitHub separately. The original PR remains open.


Current-main integration and A800 validation — 2026-10-23

Current head 4733042bd35263aab7e0c6ffabe9be0f5593b977, tree 579d40ad61dd02bca834fae6244ae2e15c3fd503, integrated with main 5ba6a3ba668cb33028eb0bce2c6d17cdc53aebe7 through normal merges. Original history is retained. Includes refreshed #113 and canonical #109. Admission uses requested units versus charged units, with large-cost drop-order regressions preserved. Graph/fixed/shared/auto modes and complete-request runtime ownership remain coupled.

Source review found a stale constructor-test error substring after #99's guard wording change; both assertions were repaired with the production guard retained. The documented Graph selector and duplicate Decider navigation were also fixed. Initial failure and focused/full repaired results are preserved.

Current-source local checks: 307 Rust workspace tests passed,0 failed,57 optional ignored; all format/strict Clippy/release/Python/contracts/smoke/strict-docs/whitespace stages passed. Four Python suites total163 tests on this head. Release runtime37 and isolated startup4 also passed. Across both heads all26 configured local stages passed; optional workspace skips are not device passes.

**Fresh A80080GB/sm80 qualification:**13 selected GPU Rust executions plus2 GPU-hidden contract executions on this head. Default and cuDNN libraries were freshly compiled as ABI9, capability/dependency receipts checked, and consumers rebuilt separately. Fresh pinned reference20 requests/56 rows, all six modes120 parity records on this head, original protocol2 numerical/schema gates unchanged. Maximum probability/per-level-fit drift0.0138, Score drift0.04. Shared/fixed logits+responses and long-auto/fixed are exact with real sharing; matched packed4/Graph4 responses are exact with positive captures/replays and zero unexpected fallbacks. No equality across differently packed controls or performance claim.

Six-mode HTTP162 logical checks,12 concurrent requests per mode with observed client overlap/direct-proxy body parity; six projection-failure retirement modes with503/unavailable health/two later refusals and sampled model memory release;18 capture-fallback requests all passed on this head. All42 genuine-model runtime scenarios passed (seven in each mode): over-budget422/no Retry-After, exact budget, request/token contention503/Retry-After1, upload/active/queued504, retained capacity until real work drains and successful same-request recovery. Full literal HTTP retained; no injected executor delay. The combined serial campaign has60 successful stages; every process stage ends at0MiB device use.

Independent agent source review inspected both complete effective diffs and all repairs, with no unresolved actionable source finding. The separate offline evidence review recomputed retained raw results and checked full source identity. This is author-side agent review, not maintainer approval or a new human attestation. Current gross diff56 files includes unmerged113/109 dependencies; net above113 is19 files,+1832/-47. Keep/split rationale and the component review map are published; review99+112→113, then109+113→149. Existing contributor-intent declarations and self-review history below are preserved verbatim; earlier sections remain historical evidence for their own source pins.

CI success · Docs success at this exact current head.

Full report, protocol, raw source/CPU/GPU/HTTP evidence, independent reviews and portable audit. All21 public assets and their checksums were downloaded and verified; downloaded audit passed. The GPU evidence was downloaded, hashed and audited before platform shutdown; fresh SSH became unavailable after shutdown. Both original PRs remain open and unmerged; main was not changed. Scope is bounded correctness on one A800, not all optional tests, broad task quality, other models/hardware, throughput, latency or peak-memory profiling.


PR113 compatibility integration — 2026-10-22

Current head: 428f834fad970f7e943180fb749871e04cbdb3e0, tree 1d68e3c9c338141aa3efe35740f996e5b22de8ad. Original149 cb5dd109 and113 5e340cab histories are retained. Only this original source branch was updated; #113 remains unchanged. No GitHub PR was merged.

Graph/fixed/shared/auto ABI9 modes now share the admission/deadline configuration and complete-request lifecycle. Both health feature sets and documentation are preserved; four isolated startup tests cover mode/runtime validation before loading. Graph+prefix remains incompatible. Current ABI is9; older sections below describe their explicitly pinned historical heads.

The effective combined change versus main@32bdd2a is 64 files +4808/-275 (above #113:20 files +1795/-45). The evidence review map separates44 files exact to113,10 exact to149 and10 combined/new. This newly combined head needs fresh review; older approvals do not attest to it. Existing human declarations are preserved without adding a new human attestation.

Validation: local305 passed/57 optional ignored, formatting/Clippy/release/strict docs and80 Python tests passed. A800:71 CPU/doc +4 startup +15 targeted device/checkpoint executions;20 reference cases/120 mode records; six-mode HTTP,42 lifecycle scenarios, six retirements and18 capture-fallback cases all passed. CI · Docs are green.

Evidence, full scope, raw retained failures, exact hashes and portable audit. All five downloaded assets and11,886 public-file entries verified; downloaded audit passed. Existing ABI9 CUDA libraries reused with exact20-file identity and fresh Rust consumers. Numerical gates unchanged; no performance claim. Earlier qualification sections below remain historical source-specific records.


Current-head A800 follow-up — 2026-10-22

Exact unchanged source: cb5dd109022aa8fae31e0fac2fe10508edfcf38f. Detailed results and limits · raw evidence and portable audit.

Targeted CPU checks passed 46 tests in each of debug and release. Fresh genuine reference/native comparison passed 18 requests / 41 rows, with maximum probability drift 0.0138 and Score drift 0.04. All seven real HTTP runtime scenarios passed: permanent overflow 422, temporary contention 503, upload/active/queue timeout 504, retained capacity and recovery. All 340 observations / 680 literal HTTP files were reparsed; every small recovery response and all 64 completed heavy answers match fresh native results.

All five published assets were downloaded and verified by bytes and checksums. The downloaded offline audit and independent review passed, including 2,488 separately obtained source-file hashes across the four heads. These current-head checks supersede earlier device-unverified statements only for the listed paths; historical results retain their original source and ABI scope. Large reproduction inputs are explicitly omitted, while measurement logs are retained. No benchmark samples or performance, broad quality or other-hardware claim are added. Existing human declarations and maintainer approval remain separate. No PR merge was performed.


Permanent-oversize response and main integration — 2026-10-22

Current head: cb5dd109022aa8fae31e0fac2fe10508edfcf38f. A processed request larger than the entire configured token ceiling cannot succeed unchanged on retry. It now returns JSON 422 (request exceeds processed-token capacity) without Retry-After; temporary request/token contention retains JSON 503 + Retry-After: 1. Both paths reject before executor dispatch and keep shared refusal accounting. A request exactly at the token ceiling remains admissible when capacity is free.

Normally merged current main 32bdd2aade0e0f3f0452d04ca5022fcaaa4783c9. Resolved the three documentation/navigation conflicts additively, retaining current JEMM, Cua-S1 screenshot/parity and Decider pages. The worker recipe now identifies the integrated Qwen ABI 8 requirement. The complete effective PR is 19 files (+1641/-36), including 1454 changed Rust lines (production and tests), below the large-change threshold. The focused post-merge increment is four files (+112/-11).

Observed regression and personally executed checks

The new HTTP regression failed against the previous response mapping (503 observed versus required 422), then passed after the focused fix. It covers idle and contended permanent oversize, JSON/content type, absent retry header, one refusal count, retained request depth, exact-ceiling admission and recovery. A separate regression preserves transient 503/retry behavior and successful retry after release. Existing deadline, cancellation, error/panic and owned-permit tests remain passing.

On the exact committed source, macOS arm64 / Rust 1.98.0:

cargo fmt --all --check
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
cargo test --workspace --locked --features omni-laya/serve
cargo build --workspace --release --locked --features omni-laya/serve
cargo test --release --locked -p omni-runtime
cargo test --release --locked -p omni-decider-native --test runtime
cargo test --release --locked -p omni-decider-native --lib serve::runtime_tests
python -m mkdocs build --strict
git diff --check

Results: 290 workspace tests passed / 0 failed / 50 ignored, including two doctests. Targeted release checks: 46 passed / 0 failed / 0 ignored (Runtime 36 including two doctests, Decider configuration/runtime 4, HTTP control 6). The separately run frontend suite passed 12 cases; those overlap workspace tests and are not added to its count. Formatting, strict Clippy, workspace release build, strict docs, relative links/examples, and diff checks passed. Python benchmark/Decider/CUDA-checker/backend suites passed 58/16/68/8 cases respectively; smoke validates four requests. CUDA text contract: zero errors, one existing Laya ABI declaration warning.

Independent agent review of the entire effective diff found no actionable finding and independently reran Runtime 34 integration tests + 2 doctests, six HTTP tests and four Decider runtime tests. The reviewed full-diff SHA256 is 4854da6fd41018f3fde42446a6f2ac26e720deb5f0ac2db509ce7c4708d03fb4.

No GPU/checkpoint execution or numerical/performance qualification was repeated. The 50 ignored tests remain unverified; historical d463ed54 ABI6 device evidence remains pinned to that source and does not qualify this ABI8 main integration. No model math, kernels, CUDA ABI definitions, lockfile or numerical gates were changed by the focused response fix. Exact-head hosted CI 39706936782 and Docs 39706936689 passed: Rust, benchmarks, CUDA contract and documentation build; PR documentation deployment was intentionally skipped.

#109's original dependency commits remain in the ancestry, together with this PR's existing request-only unit-accounting correction; #109 itself was not modified. Maintainer review/dependency ordering remains separate from these tests. Existing contributor-intent answers and self-review declarations are retained below without a new human attestation. Historical 503 descriptions apply to their named revisions and are superseded by this heading for the current oversized-token response.


Request-only counting repair — 2026-10-21

Current head: 6129a0cfec1b25f990baf9f67f937f90900be868; previous head: 06e32caa39a8cd1a0622803c6a4302d8a7782892.

Addresses the #109 request-only accounting finding and its #149 dependency handoff. When no unit budget is configured, admission/release now skip the shared unit total while each owned permit retains its original cost. Configured-unit admission, oversized refusal, exact-once release, request capacity, cancellation/deadline ownership and the declared HTTP status mapping remain unchanged. This carries the focused correction directly in this PR; #109 itself was not changed or merged.

Observed failure and regression

Applied only the two new public-API regression tests to the previous head before changing production source. With BoundedAdmission::new(2, None), retaining costs usize::MAX and 1 made both tests fail:

  • Debug: attempt to add with overflow in admission.
  • Release: a permit releases the units it was admitted with during release after the total wrapped.

Both tests now pass in debug and release. They exercise opposite permit-drop orders, preserve Permit::units(), check request-capacity refusal/rejection accounting, verify depth 2 → 1 → 0, and confirm capacity can be reused. The extreme costs reproduce the shared admission API issue; Decider's request admission still uses cost 1 and token admission still has a configured ceiling.

Personally executed CPU validation

Environment: macOS arm64, Rust/Cargo 1.98.0. All commands ran on the exact committed source:

cargo fmt --all --check
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
cargo test --workspace --locked --features omni-laya/serve
cargo test --release --locked -p omni-runtime -p omni-decider-native
cargo build --workspace --release --locked --features omni-laya/serve
git diff --check
git merge-tree --write-tree origin/main HEAD

Results: formatting, strict Clippy, workspace release build and diff checks PASS; workspace tests 228 passed / 0 failed / 38 ignored, including two doctests; release Runtime/Decider tests 57 passed / 0 failed / 7 ignored, including two doctests. Existing overload, slow-upload and preparation deadlines, expired queued work, cancellation before dispatch, retained capacity after dispatch, errors/panics and recovery tests pass. Merge preview against live main e66ff34996760a84ad56690f0ae4cf434e669e65 is conflict-free.

New-head hosted verification is separate from local checks: CI 39581129509 and Docs 39581129260 both completed successfully at 6129a0cf. Rust, benchmarks, CUDA contract and documentation build passed; PR documentation deployment was intentionally skipped.

Agent self-review covered the complete PR diff and this focused increment, ownership/callers, root-level test placement and evidence claims; no further actionable finding was identified. Increment: 2 files, +55/-12. Complete PR: 19 files, +1,542/-38, including 1,357 changed Rust lines (production and tests), below the large-change threshold. Existing contributor-intent answers and self-review declarations are retained without adding a human attestation.

No GPU/model/parity/performance campaign was run for this host-accounting repair. Device/checkpoint tests were ignored, not passed. Historical GPU evidence remains pinned to d463ed54 and its original scope; it is not a new-head qualification. CUDA/model implementation, ABI, lockfile and all files outside the two-file increment are unchanged from 06e32caa. No PR was merged; #117 remains HOLD and unmerged.

The sections below retain their named historical revisions and validation scope.

Historical documentation follow-up — 2026-10-18

Current head: 06e32caa39a8cd1a0622803c6a4302d8a7782892.
GPU-qualified source remains d463ed544bc0f6063f1513c3a72fcca2e60b8d5f; this follow-up changes only five documentation/navigation files (+191/-15). All other tracked file bytes, including runtime/model code, tests, CUDA ABI and Cargo.lock, are unchanged. The original source-specific evidence below retains its recorded revision and scope.

Addresses the latest review:

  • M1: updated src/runtime/README.md and docs/architecture.md to describe shipped request/token capacity, ExecutionContext, cancellation guards and run_with_context, including worker configuration/processor cost ownership versus shared capacity accounting and FIFO execution admission. Removed the obsolete planned-only claims.
  • L1: added recipe/decider/README.md and wired it into the recipe index and MkDocs navigation. It documents pinned artifacts, matching ABI6 build/start commands, a maintained request example, all three runtime limits, conditional health fields, strict deadlines and capacity retained by detached work.
  • L2: kept the reviewed runtime: optional bounded admission #109-compatible HTTP status mapping. The operator recipe explicitly warns that a request above the total token ceiling also receives 503 + Retry-After: 1 and cannot succeed unchanged; reduce the request or change the budget rather than blindly retrying. Switching that case to a new 4xx response is a separate API-contract decision, not a correctness requirement in the review. HTTP header names are case-insensitive; no casing-only code change was needed.

Personally executed by the agent on this documentation revision:

python -m mkdocs build --strict --site-dir /tmp/system1-pr-feedback-20261018/site
cargo test --locked -p omni-runtime -p omni-decider-native --lib --test runtime
cargo test --locked -p omni-runtime
git diff --check
git merge-tree --write-tree origin/main HEAD

The documentation build used the pinned docs/requirements.txt in an external Python 3.12 environment. Results: strict docs PASS; 47 tests passed (including two doctests), 0 failed, 2 GPU tests ignored. Checked every relative link target in the changed Markdown, all three shell blocks with bash -n, the example JSON against the exact maintained warmup input, and the generated recipe page/configuration/anchor link. Verified the entire non-documentation tree is byte-identical to the prior head. Merge preview against live main e66ff34996760a84ad56690f0ae4cf434e669e65 is conflict-free.

This follows the contributor's explicit 2026-10-18 instruction to repair review findings on the existing PRs, reusing the still-applicable contributor answers below. The agent reviewed the complete follow-up and affected implementation; existing self-review declarations are preserved, with no new human attestation. Full workspace Clippy/build/tests and GPU inference were not rerun locally for this docs-only delta. New-head hosted checks: CI, Docs; both completed successfully at this exact head. The CI run passed Rust formatting, strict workspace/all-target Clippy, frontend and workspace tests, release build, benchmark helpers and CUDA contract checks; the Docs build passed and PR deployment was intentionally skipped. The historical hosted/GPU results below are not claimed as new-head executions. No PR or dependency was merged; #117's explicit hold remains in force.

Purpose

Decider workers can opt into bounds on outstanding requests and prepared tokens, plus a deadline covering upload, preparation, queueing, execution and response reconstruction. Overload returns JSON 503 with Retry-After: 1; deadline expiry returns JSON 504. Unset limits preserve unbounded HTTP serving and existing successful responses.

Request capacity is reserved before reading the body. Token cost is the sum of complete prepared rows, including repeated state and isolated Score levels; API token usage remains unchanged. Shared execution contexts retain the original #109 owned permits across detached blocking preparation and execution. Deadline/cancellation checks run on the blocking thread immediately before work, including work queued in Tokio's blocking pool. Dispatched CUDA execution retains capacity until existing stream synchronization completes.

The Decider HTTP adapter, model preprocessing, calibration, row batching, CUDA kernels and ABI remain separate. The original SerialScheduler::run path is unchanged. Internal Engine::predict remains the direct inference/warmup API; these controls apply to HTTP serving. Token-only policy has no finite request capacity to publish; request depth/capacity/rejections are exposed when the request-count bound is configured.

Dependency and review scope

Depends on #109. Original head 56ce14a3746f5655ffe85aec8137a1689668d034 is merged without rewriting its commits or duplicating its admission implementation. Its admission.rs and admission tests are unchanged.

  • Main: f2d3d32897c1fe190fecf853a82a9456d3791462.
  • Main plus runtime: optional bounded admission #109 baseline: 9412f710 (full SHA in the evidence).
  • Integration/GPU-tested head: d463ed544bc0f6063f1513c3a72fcca2e60b8d5f.
  • Review the integration commit separately: 12 files, 793 additions / 23 deletions. The original integration diff also includes runtime: optional bounded admission #109: 14 files, 1,331 changed text lines total. The documentation follow-up below brings the current complete diff to 19 files, +1,499 / -38 (1,537 changed lines); it adds no authored code.

Prefer landing #109 first, then refreshing this branch so the dependency increment leaves the PR diff. No dependency was merged upstream by this work. This change does not include the separate Graph/prefix PRs or the held shared-vision stack.

Committed files contain core source/manifests, maintained root-level regression tests and the runtime/operator documentation added by the review follow-up. Plans, logs, measurements and generated reports are external evidence. Cargo.lock is unchanged; no dependency version or CUDA ABI changes.

Configuration / reproduction

Use the existing pinned Decider artifacts and rebuilt Qwen CUDA library:

DECIDER_MODEL=/path/to/pinned-decider \
DECIDER_CUDA_LIB=/path/to/libqwen3_5_cuda.so \
DECIDER_MAX_PENDING_REQUESTS=8 \
DECIDER_MAX_PENDING_TOKENS=1048576 \
DECIDER_REQUEST_TIMEOUT_MS=5000 \
  target/release/omni-decider

All three variables are optional. Explicit values must be positive integers; invalid configuration fails before model loading. Requests include preparation, waiting and running work. Prepared-token admission follows preprocessing. Dropping the HTTP future signals cancellation; cancelling the wait does not free resources still owned by blocking work. This PR does not add full worker lifecycle/drain or cross-request batching.

Test plan and results

Personally executed on the frozen head:

cargo fmt --all --check
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
cargo test --workspace --locked --features omni-laya/serve
cargo build --workspace --release --locked --features omni-laya/serve
  • macOS arm64 workspace: 226 passed including two doctests, 0 failed, 38 ignored. The 14 new runtime/configuration/HTTP tests also passed independent review runs.
  • Benchmark smoke: four requests validated; benchmark helpers 23, Decider Python helpers 16, CUDA backend helpers 8, backend checker tests 68 passed. Backend contract checker: zero errors and one pre-existing Laya declaration warning.
  • Exact-head fault-removal probe: deleting the context checks makes three deadline/cancellation regressions fail; the unchanged source passes all six shared control tests.
  • Separate Linux release build/CPU component tests passed. Source/archive verification covered all 546 baseline and 551 candidate tracked files.

Exact-head hosted checks completed successfully: CI 38880548987 (Rust, benchmarks and CUDA contract) and Docs 38880549001. Documentation deployment is intentionally skipped for PRs.

Demo / evidence

Report, sealed raw HTTP/GPU records, frozen protocol/helpers, source/binary hashes and local check logs.

One exclusive RTX 4090 24GB, driver 595.71.05, CUDA 13.0.88, Rust 1.98.0; pinned Decider checkpoint 533964dae8be954c5b5e19fa4948e48408094c1e, verified sizes/hashes, independent baseline/candidate Cargo targets and the same rebuilt ABI6 library. Graph/prefix modes disabled.

  • 100 measured HTTP parity requests: five configurations × concurrency 1/4 × two passes × five synthetic cases. All 60 candidate comparisons match the corresponding batch1/4 baseline status, content type and complete response bytes. Five feasibility requests are separate.
  • 10 runtime POST checks passed: request/token overload and recovery; slow upload, active request and queued request timeout; capacity still held after timeout, depth 1 then 0 after completion, and baseline-identical recovery. Two health snapshots are retained.
  • Four selected GPU tests passed: both selected-head regressions and complete eager/packed request ordering/validation-recovery cases.
  • Independent code review and independent recomputation of raw responses, corpus hashes and complete source manifests found no blockers. All 36 sealed files were downloaded and SHA-verified before the GPU server was shut down.

These are correctness checks on one GPU and a five-case corpus. No speedup, sustained-load, peak-memory, broad model-accuracy or other-hardware claim. HTTP runtime cases do not instrument individual kernel launches; synchronization guarantees also rely on source review and deterministic control tests. Aggregate token accounting is covered by CPU tests; HTTP token cases exercise oversized-request rejection. The maximum-row GPU regression and other ignored device/checkpoint tests were not run in this campaign. Video: N/A for this nonvisual runtime change.

Required human contributor answers

Source: the contributor's explicit 2026-10-16 development-task instruction, selecting Decider runtime admission/deadline integration based on #109 and directing development through a core-only PR. Private connection details are excluded.

  1. Outcome: implement and submit Decider runtime admission and deadline integration.
  2. Chosen variable: the selected runtime request/token capacity and deadline behavior.
  3. How the work moves it: integrate existing runtime: optional bounded admission #109 into the real worker, avoid a second admission implementation, and validate the complete request lifecycle.
  4. Customer/need: the requesting framework contributor, who asked for this scoped development through a complete PR with core changes only.

PR needed? Yes; the contributor explicitly requested continuing until the PR is submitted. This is an opt-in feature, not a claim that unbounded serving was an existing numerical bug. Fault-removal regression evidence is linked above.

Self-review

Agent-executed checks and independent review under the contributor's development instruction; these statements do not attest personal human review or replace maintainer review.

  • Agent reviewed the complete integration and dependency diff and addressed findings.
  • Agent checked request/processor/scheduler/executor ownership and resource lifetimes.
  • Agent ran relevant CPU/GPU checks and disclosed ignored tests and evidence limits.
  • Agent audited the core-only diff and independently verified the published evidence against the frozen head.

xiaoyu-xyz and others added 5 commits October 11, 2026 13:15
`SerialScheduler` admits one unit at a time and never refuses: a hundred requests
queue behind each other and the only bound is the caller's patience. That stays
the default. This adds what a worker opts into when it wants a bound instead,
which is the first step review pointed at — the bounded admission, with the
worker lifecycle left out.

Units are the caller's and they are explicit. Admission counts how many requests
are outstanding and what each one costs; a request costs one, and the cost is
whatever the caller passes. A prefill worker can admit by token count, a worker
with large payloads by bytes, and a worker that wants neither by `1`. Nothing
here guesses which of those a model means, which is why the cost is a parameter
rather than a documented convention.

The permit is the reservation. Capacity is held until the permit drops, so
"outstanding" means "accepted and not yet finished with" rather than "accepted".
A worker that hands work to a thread and forgets to release would be bounded
forever, so release cannot be something a caller has to remember. The permit is
deliberately neither `Clone` nor `Copy`: two permits for one admission would
release capacity twice, which is the failure worth preventing.

A request costing more than the whole unit ceiling is refused rather than
admitted and overrun, which a plain `depth < capacity` check gets wrong. A zero
bound panics at construction rather than showing up as every request being
turned away.

Not here, per the same guidance: worker-thread ownership, readiness transitions,
drain on shutdown, and any change to `SerialScheduler` or to the workers. This
changes no existing behaviour — nothing is bounded until something asks.

Eight tests under the repository `tests/` tree cover the request bound, release
on drop, unit costs, the oversized single request, the reported capacity, both
zero-cost refusals, and shared clones.
The fit check compared with `saturating_add` while the reserve added
directly, so the two disagreed for a unit cost near `usize::MAX`: the
check could pass and the reserve then overflow. Both now compute the sum
the same way.

Found by review on InferMatrixLab#109 (P3, `src/runtime/src/admission.rs:97`).
Two findings from review of InferMatrixLab#109, both at this type's seams.

**A permit can be moved into the work it was granted for.** It held a
`&BoundedAdmission`, so it could not go into the runtime's own blocking
executor, which needs `'static` work: `spawn_blocking(move || ...)` with a
borrowed permit does not compile. `BoundedAdmission` now wraps a shared
inner, `Permit` owns a share of that inner, and `try_admit` takes `&self`
and hands back something that outlives the handle that issued it.

**The counter cannot be driven negative.** The permit stored the units its
caller asked for and released that, while the counter had been advanced by
a clamped value; where the two differed, a later release underflowed and
panicked. The permit now stores the units it actually reserved. The
clamping is gone with it: a request costing more than the ceiling is
refused outright, so a reservation is never larger than the budget it was
admitted under and the held total cannot reach `usize::MAX` by
accumulation.

Tests, including the three gaps the review named: concurrent admit/release
over 32 threads for 100 rounds asserting the live count never passes the
bound, recovery after a panic while the counters are locked, a permit moved
into another thread and released there, and the ceiling cases that used to
unbalance the counter.
Copilot AI balanced review requested due to automatic review settings October 16, 2026 09:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review — PR #149 "decider: bound pending requests and enforce HTTP deadlines"

Reviewer: independent code review (ZCode agent). Not posted to GitHub; no repo modifications. Local review only.

Provenance

Item Value
Base (live origin/main) e66ff34996760a84ad56690f0ae4cf434e669e65 (merge of #116, benchmarks/serving-matrix)
PR head (live, re-fetched before report) d463ed544bc0f6063f1513c3a72fcca2e60b8d5f (branch codex/decider-admission-deadline, OPEN, non-draft, author Levius-Fubuki)
Merge base f2d3d32897c1fe190fecf853a82a9456d3791462 (merge of #111) — the PR is forked from pre-#116 main
Merge cleanliness git merge-tree --write-tree origin/main origin/pr/149 succeeds; #116 touched only benchmarks/* + tests/benchmarks/test_serving_matrix.py, zero file overlap with this PR
Skill load .agents/skills/system1-omni-review/SKILL.md read at the trusted local revision; SHA-256 of bytes read: 58be3bc60806c4c5af7038a0c6a80fadb94f3360d714556a3e965c8903dde108 (matches expected)
References read docs/architecture.md, src/runtime/README.md, .agents/skills/system1-omni-review/references/repository-contracts.md, CONTRIBUTING.md, PR body — all at origin/pr/149
Host constraints honored No Rust toolchain on this host (cargo not invoked); no GPU. Rust checks are covered by observed CI (below).

Note: the skill references CONTRIBUTING.md#large-code-changes, but CONTRIBUTING.md at both origin/main and the PR head contains no such section (headings verified). The PR is below any plausible large-change threshold anyway, and the author supplies the split regardless.

Diff inventory and authored/total split

Full PR diff vs merge base: 14 files, +1308/−23. Decomposition verified by per-commit diffs:

  • Inherited #109 prerequisite (merge 9412f71): src/runtime/src/admission.rs (+208), tests/runtime/admission.rs (+302), src/runtime/src/lib.rs (+1), src/runtime/Cargo.toml (+4) = 4 files, +515.
  • Authored integration commit d463ed5: 12 files, +793/−23 — decider runtime.rs (new, +126), serve.rs (+121/−23), engine.rs (+26), executor.rs (+13), lib.rs (+1), Cargo.toml (+9); runtime control.rs (new, +73), lib.rs::run_with_context (+21), Cargo.toml (+4); tests tests/decider/runtime.rs (+69), tests/decider/serve_runtime.rs (+118), tests/runtime/control.rs (+235).
  • The #109 content is byte-identical to approved #109 head 56ce14a (git diff 56ce14a origin/pr/149 -- src/runtime/src/admission.rs tests/runtime/admission.rs is empty). "Unchanged prerequisite" claim verified.
  • No generated artifacts, data dumps, logs or plans committed. Cargo.lock unchanged despite the dev-dep addition (tokio io-util already in the lockfile graph). #[path]-included tests/decider/serve_runtime.rs follows the existing declared-target pattern (batching.rs etc. already declared [[test]]); it is a maintained test, not run output.

The PR body's stated split ("12 files, 793 additions / 23 deletions" integration; "14 files, 1,331 changed text lines total") matches my computed numbers exactly.

Findings

Medium

M1 — src/runtime/README.md (the normative runtime contract) is now stale relative to shipped capability. The contract still says: "Queue-length limits, token budgets, compatibility grouping, shared processing orchestration and GPU batching remain planned." This PR ships opt-in queue-length limits and token budgets (admission::BoundedAdmission) and adds a second new public module (control::ExecutionContext / ExecutionStopped / CancellationGuard) plus a new public SerialScheduler::run_with_context, none of which the contract describes. The relationship between the scheduler's existing execution-unit admission (SerialSemaphore, one unit) and the worker-level BoundedAdmission request/token budgets — two coexisting admission layers — is documented only in module doc comments. docs/architecture.md also says "the scheduler owns queue and admission policy rather than duplicating it inside each model implementation" (scheduling contracts), while the policy entry point (env parsing, admit/reserve sequencing) now lives in the model worker (src/models/decider/native/src/runtime.rs) with the shared type in the runtime — defensible, but exactly the boundary the contract README should state. No behavior bug; a contract-update requirement before or with merge. Note the decider's own module doc in admission.rs/control.rs is otherwise excellent and internally consistent.

Low

L1 — New operator-facing configuration is documented nowhere a user looks. DECIDER_MAX_PENDING_REQUESTS, DECIDER_MAX_PENDING_TOKENS, DECIDER_REQUEST_TIMEOUT_MS and the 503/504 semantics exist only in Rustdoc and the PR description. There is no decider worker README; recipe/decider/ contains only download_weights.py; the decider worker is not mentioned anywhere in docs/ or the top-level README (verified by git grep). Repo pattern documents worker opt-in env behavior in docs/recipes (e.g. CUA_S1_GRAPH=1). Suggest a recipe/decider/README.md (serving, limits, deadline semantics, health fields) as a fast-follow; not a merge blocker for a worker that is already undocumented as a whole.

L2 — A permanently inadmissible request is answered 503 + Retry-After: 1. A request whose processed_tokens() exceeds the entire DECIDER_MAX_PENDING_TOKENS ceiling can never be admitted, yet it receives the transient-overload response (503, "inference capacity exhausted", Retry-After: 1), which invites deterministic retry loops from frontends that honor Retry-After. A 4xx (413/422, "request exceeds configured token budget") would distinguish request-property failures from transient capacity. This follows #109's reviewed refusal→503 mapping and is disclosed in the body, so it is a design note, not a contract violation. Cosmetic sibling nit: the header is inserted lowercase as "retry-after" (valid HTTP/1.1, unconventional casing).

Info / maintainer awareness

  • I1 — Strict post-execution deadline check discards completed work. On the success path, infer re-checks the context after finish(rows); if the deadline expired between completion and response, a fully computed body is discarded for a 504 (serve.rs success arm). Deliberate strictness, covered by tests, but it converts "just missed" into "all cost paid, nothing delivered." Fine if intended; flagging so it is a decision rather than an accident.
  • I2 — Deadline expiry abandons but does not preempt. timeout_at drops the request future (releasing permits via shared Arc<Permit> and setting cancelled only when the whole handler scope ends), while detached blocking prepare/execution runs to completion and retains capacity until then; queued followers wait behind the doomed unit. This matches the architecture contract ("Once dispatched, the runtime retains admission and captured executor resources until the blocking closure completes") and is disclosed in the body and encoded in tests. Absolute-deadline checks at every boundary (pre-queue, closure start, post-prepare, post-execute) stop anything that has not started.
  • I3 — Health metrics are conditional. /health publishes depth/capacity/rejected only when a request bound is configured; token-only deployments publish no queue metrics (documented in code and body). /health itself is not admission-gated — correct per repo pattern.
  • I4 — Author-reported human-gate provenance. The post-#138 template's "Required human contributor answers" cites "the contributor's explicit 2026-10-16 development-task instruction" as the answer source. I cannot verify a private instruction; reported as author-asserted with a traceable-source claim, which is what the template requires.

Per-focus analysis

1. BoundedAdmission integration — conforms to approved #109 and the runtime contract.

  • The worker uses the approved type unchanged (byte-identical to 56ce14a): try_admit → owned Permit, released exactly once via Drop (Permit is deliberately not Clone/Copy; ExecutionContext::retain wraps in Arc<Permit> so clones share one reservation); oversized costs are refused, never clamped (units > ceiling || held.units > ceiling - units, with a dedicated usize::MAX-cost test); reservations store what was actually reserved; poisoned-lock recovery; checked_sub guards.
  • Units: request budget counts 1 per request (acquired before body read, so it covers upload + preparation + queueing + execution); token budget uses PreparedRequest::processed_tokens() = sum of complete row token lengths, including repeated state and Score expansion (verified: it is a plain sum(), distinct from the pre-existing prefix-dedup unique_tokens). Bytes are not a unit here; body size is separately capped by the existing 8 MiB DefaultBodyLimit. Units are the caller's, per the type's contract.
  • Permit lifetime through preparation AND execution: the request permit lives in the ExecutionContext that is cloned into the detached spawn_blocking prepare closure and into run_with_context's blocking closure (drop(context) after work()); the token permit is acquired post-prepare and follows the same path. Client disconnect or 504 abandonment drops the async clones while the blocking closure retains its clone until the work ends — the #129 omnijev lesson is directly implemented and tested (cancellation_after_dispatch_keeps_both_reservations_until_work_finishes: after task.abort(), both admissions still held; deadline_covers_preparation_and_returns_504: depth still 1 after the 504, then 0 after the blocking work ends).
  • Cancellation before/after dispatch per contract: run_with_context checks before queueing; the blocking closure re-checks at start ("dispatch itself can wait behind unrelated blocking work" — covered by blocking_pool_backlog_checks_deadline_before_starting_work and ..._cancellation_before_starting_work with max_blocking_threads(1)); after dispatch, retention until completion, including panic/error paths (errors_and_panics_release_owned_capacity).
  • No deadlock: strict request→token acquisition ordering everywhere; single admission call sites (Runtime::admit, Runtime::reserve_tokens).

2. HTTP deadlines.

  • One deadline, started + DECIDER_REQUEST_TIMEOUT_MS, measured from admission; enforced by tokio::time::timeout_at wrapping the entire request future — body upload included (verified with a real socket test: declared Content-Length 100, one byte sent → 504, depth back to 0) — plus absolute-deadline check()s at every cooperative boundary. There is no separate queue-wait deadline; queue wait is inside the same envelope, and a queued unit whose deadline expires while sitting in Tokio's blocking pool is rejected at closure start without executing.
  • Expiry mapping follows #109's reviewed pattern: refusal/overload → JSON 503 + Retry-After: 1 (Overloaded); deadline expiry (timeout branch or ExecutionStopped::Deadline) → JSON 504. Errors are typed (Overloaded, ExecutionStopped implement std::error::Error), envelope {"detail": ...} consistent with the existing worker helper; 415/413/422 paths preserved (413 exercised by test).
  • Expiry abandons the response and cancels cooperatively; it does not preempt in-flight blocking/device work; permits release exactly once in both the abandon case (async clones drop; blocking clone drops at work end) and the natural-completion case. See I1/I2.
  • Deadlines are NOT documented in any recipe/doc (L1); they are documented in code, the health metadata (timeout_ms under runtime_limits), and the PR body.

3. Interaction with the decider's existing paths (#111–#113 content).

  • Admission wraps the whole decision uniformly: decide → content-type check → readiness check (both before admission; a non-ready or malformed request consumes no slot) → within_budget (admit) → read body → infer (prepare outside the model mutex → reserve_tokens → execute_with_context). There are no per-mode differences to worry about: the native decider is eager-only and rejects graph mode at load ("Decider eager execution requires CUA_S1_GRAPH unset or 0", executor.rs:202), so there is no Graph capture path a deadline could fire mid-capture. The legacy Executor::execute is preserved unchanged (delegates with a default, deadline-free context), and SerialScheduler::run itself is untouched, so #111/#113 behavior for non-HTTP callers (warmup, Engine::predict, batching tests) is unaffected.
  • The token reservation happens after preparation, so a token-bound worker does preparation work for requests it then refuses — unavoidable at this granularity, bounded by the request budget.

4. /health and error envelope. /health is not admission-gated (repo pattern) and now additionally reports depth/capacity/rejected when a request bound is configured (I3). Envelope/status mapping consistent; see focus 2 and L2.

5. Tests.

  • Concurrency: the #109-pattern test is present (concurrent_admissions_never_exceed_the_bound_and_all_come_back: 32 threads, Barrier, 100 rounds, capacity 4, peak and leak assertions).
  • Deadline/cancel: 6 control tests (expired queued work never executes; post-dispatch cancellation retains both reservations until work finishes; blocking-pool backlog deadline and cancel checks; errors/panics release; cancel-while-waiting releases without dispatch) + 4 HTTP tests (503/Retry-After without reading the body; slow-upload 504 over a real TCP socket with capacity recovery; preparation deadline 504 with depth 1→0; success/413/415 preservation).
  • Drain/shutdown: explicitly out of scope ("This PR does not add full worker lifecycle/drain" in the body); no drain tests added. Consistent scoping, noted for the record.
  • Weight-free Python (run by me at the extracted head tree): python -m unittest discover -s tests/decider -p 'test_*.py' → 16/16 OK (0.017s). These files are unchanged by the PR; run confirms no regression at head.
  • Rust tests: not runnable here (no toolchain, per instructions). Observed CI rust job PASS at the exact head (below); author reports 226 passed / 0 failed / 38 ignored (macOS arm64, --features omni-laya/serve) — author-reported, not independently verified. The 38 ignored include checkpoint/GPU-dependent tests; author's GPU evidence is separate (below).

6. Docs, template, hygiene.

  • Docs: no documentation files changed (M1, L1). docs/architecture.md statements remain true as written (the new capability is worker-opt-in, not shared orchestration), but the runtime README is stale (M1).
  • Template readiness (post-#138): PRESENT and complete. "Required human contributor answers" — all four questions answered with a named source (2026-10-16 development-task instruction), explicit "PR needed? Yes" decision with rationale, and the non-bugfix evidence slot answered (not applicable + fault-removal probe offered). Self-review checklist: all four boxes checked, with an explicit upfront statement that the checks are agent-executed and do not attest human review — the honest framing the template wants. I did not verify the private instruction (I4).
  • Authored vs total: reported and verified (12 files +793/−23 authored; +515 inherited #109; 14 files +1308/−23 total).
  • Artifact hygiene: clean. Source, manifests and maintained root-level regression tests only; no plans/logs/reports committed; Cargo.lock untouched; evidence (sealed raw records) is external in a fork release.

Commands run and results

Command Result
git fetch origin main --force live main e66ff34996760a84ad56690f0ae4cf434e669e65 (advanced past f2d3d32 via #116, benchmarks-only)
git fetch origin "pull/149/head:refs/remotes/origin/pr/149" --force live head d463ed544bc0f6063f1513c3a72fcca2e60b8d5f (matches GitHub headRefOid)
git merge-tree --write-tree origin/main origin/pr/149 merges clean
git diff 56ce14a origin/pr/149 -- src/runtime/src/admission.rs tests/runtime/admission.rs empty — #109 content unchanged
shasum -a 256 .agents/skills/system1-omni-review/SKILL.md 58be3bc6…dde108 (match)
python -m unittest discover -s tests/decider -p 'test_*.py' -v (miniconda python, extracted head tree under /tmp) 16/16 OK
gh pr checks 149 --repo ThinkFlowLab/system1-omni rust PASS (4m48s), benchmarks PASS (11s), cuda-contract PASS (8s), build PASS (20s), deploy SKIPPED (intentional for PRs)
gh run view 38880548987 … headSha d463ed5…, pull_request, success, 2026-10-16T09:27:11Z — CI provenance pinned to the exact head; run IDs match the PR body's claims
gh release view decider-runtime-d463ed54-20261016 --repo Levius-Fubuki/system1-omni exists, 4 assets, created 2026-10-16T09:17:45Z (existence verified; contents not downloaded)

Not run: any cargo command (host has no Rust toolchain — contract honored); no GPU validation (host has none); fork release asset contents not downloaded.

Material unverified checks

  1. Rust test/clippy/build execution — covered by observed CI PASS at head, but not independently re-executed; author's "226 passed / 38 ignored" macOS run and separate Linux run are author-reported.
  2. GPU HTTP parity and timeout campaign (RTX 4090, 100 parity requests, 10 runtime POST checks incl. slow-upload/active/queued timeouts and capacity-hold-after-timeout) — release exists with 4 sealed assets; contents and results not independently verified; no GPU on this host. This is the only end-to-end evidence that deadlines behave correctly against the real executor (the CPU tests use synthetic closures), so it is the main residual evidence gap.
  3. Post-#138 human-gate source — the cited private contributor instruction is unverifiable by me (author-asserted).
  4. --features omni-laya/serve variants of the author's checks — the extra feature flag goes beyond the CI baseline; not re-verified here (CI ran the baseline commands and passed).

Verdict

No blocking correctness findings. The integration is faithful to approved #109 (type unchanged, exactly-once release, oversized-refused, slot-held-until-work-ends including disconnect/abandonment), the deadline machinery matches the runtime contract's cancellation/retention rules, the tested behaviors are the right ones (including the two regressions that encode the hard-won lessons), CI is green at the exact head, and the PR body's counts and SHA claims all check out against the live revision. Required before/with merge: update src/runtime/README.md so the contract no longer describes shipped queue-length/token budgets as "planned" and states the two-layer admission relationship (M1); recommended fast-follows: operator docs for the three env vars and overload/timeout semantics (L1), and reconsider 503/Retry-After for permanently oversized requests (L2). Scope is coherent (no drain, no batching, graph modes N/A in the native worker), and template readiness is complete with honest agent-execution disclosure.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-check at new head 06e32caa39: one new commit since my review at d463ed544b — "docs(decider): explain runtime admission and HTTP limits" (5 files, +191/−15, docs/mkdocs only); the four reviewed commits are patch-identical (git range-diff, named refs, cat-file verified).

The P2 and L1 from my review are fixed, verified in the diff: src/runtime/README.md now documents the two admission layers and their distinct lifetimes (SerialScheduler FIFO execution vs the worker's BoundedAdmission refusals), the decider's reservation semantics (request reservation covering preparation; token reservation after preparation), the costs-above-budget refusal (never clamped), and control::ExecutionContext's deadline/cancellation structure — resolving the "remain planned" contract staleness. recipe/decider/README.md (+121) and the mkdocs/recipe registrations close the worker-docs absence. Decider Python tests pass at this head; mkdocs build --strict passes (mkdocs.yml touched).

One Low remains, unchanged (code untouched by this commit): a request whose tokens exceed the whole ceiling still gets 503 + Retry-After: 1 — inviting futile retries where a 4xx would fit better. Verdict: COMMENTED (that Low). Standing unverified: Rust suite CI-corroborated only; the RTX 4090 HTTP/parity campaign author-run; evidence release contents not re-derived.

Copy link
Copy Markdown
Collaborator Author

Dependency follow-up for #109

This branch contains the same admission.rs implementation as #109, so please carry its request-only unit-accounting fix when refreshing the dependency.

The current Decider request-admission path uses cost 1, while token admission has a configured unit ceiling. The generic usize::MAX reproducer should therefore remain scoped to the shared admission API.

Keep the current ownership behavior: detached preparation/execution retains its reservations until completion and device synchronization, even after the HTTP response times out. Validate overload, upload/queue/execution deadlines, retained capacity and eventual recovery.

The documentation and status-mapping follow-up is already present. This review does not request changing the declared 503 + Retry-After contract.


Reviewed head: 06e32caa39a8cd1a0622803c6a4302d8a7782892.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-check at new head 6129a0cfec: one new commit since my review at 06e32caa39 — "fix(runtime): skip unit totals for request-only admission" (admission.rs + 39 lines of tests); the five reviewed commits are patch-identical (git range-diff, named refs, cat-file verified).

The new commit is reviewed and sound: with units: None (request-only admission), held.units is no longer maintained — removing the overflow-panic class where unbounded caller costs accumulated toward usize::MAX (the pass-50 ceiling tests exposed exactly this class). With a unit ceiling configured, tracking is unchanged (guarded increment, checked_sub release under is_some()), preserving the approved refusal-not-clamp semantics; Reservation still stores the requested cost. New tests pin the behavior: a usize::MAX-cost permit in request-only mode admits, releases, and re-admits, in both large-first and small-first orderings, plus recovery.

The Low from my review remains open and untouched by this commit: a request whose cost exceeds the whole unit ceiling still gets 503 + Retry-After: 1 rather than a 4xx. Fork CI at this exact head: benchmarks/build/cuda-contract (and rust) pass.

Verdict: COMMENTED (that Low remains). Standing unverified: Rust suite CI-corroborated only; GPU HTTP/parity campaign author-run.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-check at new head cb5dd10902: one new commit since my review at 6129a0cfec — "fix(decider): reject requests above the token capacity with 422" — plus a merge of current main (own reviewed commits patch-identical; git range-diff, named refs, cat-file verified; merge resolutions carry the absorbed jemm/decider lineage, consistent with the pass-57 wave verification).

The Low from my review is fixed, verified in the diff: a single request whose processed-token cost exceeds the entire token ceiling now returns 422 {"detail":"request exceeds processed-token capacity"} with no Retry-After header, rejected before execution — while 503 + Retry-After is reserved for temporary request/token contention (docs in README + recipe updated to the same effect, and rejected now counts ceiling-exceeding refusals). New OversizedRequest error type; a request costing exactly the ceiling is still accepted when capacity is free.

Also reviewed: the merge resolutions are docs-only unions consistent with the pass-57 wave verification of this stack.

Fork CI at this exact head: benchmarks/build/cuda-contract (and rust) green. No actionable findings remain. APPROVED. Unverified: the RTX 4090 HTTP/parity campaign remains author-run (no GPU on this host); Rust suite covered by exact-head CI; GPU execution of ignored device tests author-run.

Copy link
Copy Markdown
Collaborator Author

Current-head A800 runtime qualification for cb5dd109022aa8fae31e0fac2fe10508edfcf38f; source unchanged. Latest review5608436425 confirms the422 fix and reports no actionable findings; this comment supplies fresh device evidence rather than a new formal approval.

  • Targeted CPU runtime/HTTP checks: 46 passed in debug and46 in release, no failures/ignored cases.
  • Fresh genuine pinned author-reference/native comparison: 18 requests/41 rows passed, probability drift max0.0138, Score max0.04, no disagreements, unchanged protocol2 gates.
  • Seven real-worker HTTP scenarios passed: above-total token cost422 JSON without Retry-After; exact ceiling51 accepted; temporary request-slot and token contention503 with Retry-After1 and recovery; slow-upload504; real active/queued500ms deadlines504, retained depth1/capacity refusal while genuine work continues, eventual same-request recovery. The heavy request uses64 genuine long-context rows/226,496 processed tokens; no fake executor or injected delay was used.

All 340 observations/680 literal HTTP files were reparsed against saved status/headers/JSON/timestamps. All seven small200 bodies, including upload recovery, equal fresh native output; all64 completed heavy answers match the fresh long-context result. HTTP observations establish retained admission; the exact CUDA synchronization instant is supported by unchanged source and CPU lifetime tests, not a socket timestamp.

All625 source files, genuine pinned inputs, binaries and matching ABI8 library were rechecked; GPU residency returned0. Library reuse is backed by exact20-file build-input equality. No source change, benchmark sample, performance/quality claim, manufactured approval or PR merge.

Raw evidence, exact source/input hashes, commands/failures, portable offline audit and independent review. All five downloaded assets matched their sealed bytes/checksums; the downloaded audit passed, including2488 separately obtained source files across the four PRs. Full source/model/native payload inputs are omitted explicitly, while all measurement logs are retained. The offline audit checks recorded evidence, not independent authentication of GPU execution.

Integrate PR113 5e340ca into PR149 while preserving graph/prefix/fixed/auto controls, mode health statistics, bounded requests/tokens and cancellation-safe HTTP deadlines.

Use a shared explicit mode/runtime loader, retain existing constructor defaults, and add isolated startup configuration regressions. Keep the ABI9 backend and matching native consumers together.

Copy link
Copy Markdown
Collaborator Author

Compatibility integration with #113 completed at 428f834fad970f7e943180fb749871e04cbdb3e0, tree 1d68e3c9c338141aa3efe35740f996e5b22de8ad. The original #149 branch was fast-forwarded from cb5dd109; its normal local Git merge retains original #149 and #113 5e340cab as parents. No GitHub PR was merged. #113's source branch is unchanged.

The common Engine constructor now combines explicit Graph/fixed/shared/auto modes with runtime configuration and validates incompatible modes/invalid budgets before model loading. Every execution mode stays inside the same context-owned scheduler operation, including stream synchronization, retirement and capacity release. Graph/prefix health counters coexist with runtime limits/depth/capacity/rejected. Documentation retains both operator guides; four child-process-isolated startup regressions cover early validation. Graph+prefix remains intentionally incompatible as in #113.

Scope for fresh review: versus main@32bdd2a this combined stack is 64 files +4808/-275; versus #113 it is20 files +1795/-45. Forty-four changed files equal #113 exactly, ten equal #149 exactly, and ten are combined/new. The review map and patch distinguish inherited prerequisites from integration work. Earlier maintainer approval belongs to its named historical heads; it is not represented as approval of this new integration.

  • Local exact source: workspace 305 passed /57 optional ignored, formatting, Clippy, release build and strict docs passed;58 benchmark +22 Decider Python tests passed.
  • A80080GB/sm80: corrected CPU/doc 71 passed, separate startup 4 passed, and 15 targeted device/checkpoint Rust pass executions (mixed GPU and CPU checkpoint cases).
  • Fresh reference 20 cases, six modes 120 comparison records; fixed/shared logits+responses exact, long auto exact to fixed with positive sharing, positive Graph capture/replay. Original protocol2 gates unchanged.
  • Six-mode direct/frontend HTTP: 390 saved response bodies, including294 successful POST occurrences with exact same-mode native parity. Runtime: 42 scenarios /3,170 literal observations covering permanent422/no Retry-After, exact-ceiling200, request/token contention503+Retry-After1, upload/active/queue504, positive retained depth/capacity and recovery.
  • All six real projection-failure retirements passed (unavailable health, depth0, later refusals, released GPU state); Graph capture-failure fallback passed18 cases. Initial/final644 source hashes, model-directory entries, two ABI9 libraries and three fresh Rust consumers were rechecked; GPU memory returned0 MiB. Existing CUDA libraries are reused with exact20-file build-input identity, not claimed freshly rebuilt.

Exact-head CI · Docs are green (Rust, CUDA contract, benchmarks and docs build).

Immutable evidence, reconstruction patch, review map, raw records, independent review and portable audit. Archive 1,297,304 bytes, SHA256 36d4f60af0a405eea6266187e934267583296844866dd1283878196876baae6f;11,886 public files. All five release assets were downloaded and byte/checksum verified; every manifest entry and the downloaded offline audit passed. Both failed harness attempts (early queue probe200; token probe taking capacity before holder503) and environment corrections are retained, not counted as passes. Readiness was corrected without changing source, numerical gates,64-row workload,500ms deadlines or required outcomes. Large reproduction inputs are explicitly omitted with hashes/provenance. Zero performance samples; offline audit reviews recorded author execution, not independent GPU authentication.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-check at new head 428f834fad: the branch merged the decider stack (approved #111/#112/#113 lineage) beneath the reviewed admission/deadline work. Verified: src/runtime/src/admission.rs, lib.rs and Cargo.toml are byte-identical between the approved head cb5dd1090 and this head — the PR's own content is unchanged — and the own-commit series is patch-identical per git range-diff (named refs, cat-file verified). The absorbed decider content is exactly what I approved at 5e340cab24/#113 and 3de420ccf2/#112.

My pass-73 approval at cb5dd1090 carried zero open findings (the L2 oversize→422 Low was fixed by cb5dd109 itself, verified then). CI at this exact head: benchmarks/build/cuda-contract (and rust) pass on the fork.

APPROVED (verdict carries — own content unchanged; absorbed content previously approved). Unverified: GPU HTTP/parity campaign author-run (no GPU on this host); Rust suite covered by exact-head fork CI. Stack-order note: this branch now contains the #111→#113 stack; merge order remains with the maintainer.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

State correction: my approval of this PR at this exact head (428f834fad, pass 74) was posted with the wrong review event type (COMMENT instead of APPROVE) due to the same posting-script error — the full approval verdict and verification are in that review body directly below and are unchanged: own admission/deadline content byte-identical to the approved cb5dd1090, zero actionable findings (the oversize→422 Low was fixed by cb5dd109 itself), fork CI green at this head. This review corrects the recorded state to match that verdict. Unverified: GPU HTTP/parity campaign author-run; Rust suite via exact-head CI.

With `units: None` the request bound is the only limit, but `try_admit`
still charged the caller's cost to the shared total. That total has no
ceiling and nothing reads it in this configuration, so the charge bought
nothing and could overflow: cost `usize::MAX` followed by `1` overflows in
a debug build, and in a release build it wraps to zero and the next
`release` panics inside `Drop`, which aborts the process rather than
unwinding.

The total is now advanced only when a unit budget exists, and `release`
subtracts only then. The permit still reports the caller's cost through
`units()`, so accounting done from a permit is unchanged.

Reported by review of InferMatrixLab#109 (P2, `src/runtime/src/admission.rs:121-136`).

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #149 incremental review — pass 84 (head 4733042bd)

Verdict: APPROVED. Prior on-record state: APPROVE at 428f834fa (state-correction review, 2026-10-22T10:07Z, correcting the mis-evented pass-74 approval).
Date: 2026-10-23.

Re-pinned state (source inspected)

  • Live head re-confirmed: 4733042bd35263aab7e0c6ffabe9be0f5593b977 (git rev-parse origin/pr/149 after fetch; gh pr view 149 agrees). Non-draft, title unchanged ("decider: bound pending requests and enforce HTTP deadlines").
  • First-parent chain exactly as pinned, all objects exist (git cat-file -e OK):
    428f834fa → fd93d8d4 (merge) → 2c56521b (merge) → 362ead0c (merge) → 4733042bd (authored).
  • Own-series integrity: NO own commit rewritten. All four new commits sit on top of the previously reviewed head 428f834fa; cb5dd1090 and 428f834fa remain ancestors unchanged. Range-diff gate not triggered (no rewritten own range), so no range-diff was required.

Merge hygiene (all three merges)

  • fd93d8d4 (parents 428f834fa + 017a71b37 = approved #109 head): MANUAL resolution (merge-tree 9b6654c6 ≠ actual tree d84ef6d0). Every resolution hunk read. Both code conflicts take #109's canonical side (held.units += charged; and .checked_sub(reservation.charged)) over #149's guard-based equivalent — semantically identical (charged=0 when no unit budget). Doc comments are merged wording, verified accurate against the code.
  • 2c56521b (parents fd93d8d4 + 782db6f4 "Merge latest shared-prefix executor…"): PURE auto-merge (tree matches git merge-tree --write-tree).
  • 362ead0c (parents 2c56521b + b3cd06a2 = #113's authored commit): PURE auto-merge.

Entering content provenance (the key question)

Net delta 428f834fa..4733042bd: 16 files, +649/−58. Attribution by per-merge net diff:

  1. Via fd93d8d4 from approved #109 head 017a71b37:
    • src/runtime/src/admission.rs (+42 net): CODE byte-identical to approved 017a71b37. The ONLY divergence is the Reservation struct doc comment (lines 70–74), which replaces #109's known P3 (stale clamping description) with an accurate description of units/charged semantics — verified line-by-line against the code (refusal not clamping at line 133; charged=0 for request-only at 142; release subtracts charged once at 179). NOTE with attribution: #109's P3 is carried in name but actually RESOLVED in this tree by the merge resolution.
    • tests/runtime/admission.rs (+29): clean UNION — #149's own a_request_only_bound_does_not_accumulate_units retained AND #109's paired release-order tests (request_only_admission_releases_{large,small}_cost_first) added. No test lost; overlapping coverage only.
  2. Via 362ead0c from #113's authored b3cd06a2 (under parallel #113 review): recipe/decider/validation.md (+3/−1, pins the graph regression to ordered_shape_replay_current_ids_growth_and_bounded_cache, which exists at tests/qwen3_5/graph.rs:7) and tests/qwen3_5/graph_loading.rs (assertions "require merged weights" → "only merged weights", matching the actual message at model.rs:1661). Both files byte-identical to b3cd06a2 (empty diff). Cross-referenced, not re-reviewed in full.
  3. Via 2c56521b from the 782db6f4 side (main/#93/#98/#112 lineage): 10 files, +588/−33.
    • Byte-identical to current origin/main (canonical main content): benchmarks/README.md, benchmarks/bench.py (+--score-decimals, Decider zero-fit exception, exec bit restored), tests/benchmarks/test_score_wire.py (+225), src/backends/cuda/contract.md, src/backends/cuda/qwen3_5/gemm.cu.
    • contract.md example "abi_version": 7→4: aligns the PR with current main (origin/main has 4; the 7 was the side-lineage anomaly at 5e340cab). Real manifest is 9; the example's staleness is pre-existing on main, not introduced here.
    • Net-new (not on main) side-branch content, reviewed in full: tests/qwen3_5/model.rs (+226 — child-process profile-refusal test with compiled fake CUDA library; sound), tests/qwen3_5/profile_fake.c (+20 — CPU test double), gemm.cu reduction-scheme selection relocated into a ternary in plan_for (semantics-preserving), src/models/qwen3_5/native/src/model.rs (ensure_fixed_profile() extraction with reworded error message; PR-side ahead of main with consistent docs).
    • docs/architecture.md, src/backends/cuda/qwen3_5/README.md: PR-side ahead of main (Decider paragraphs); only the small merged-weights sentence/ABI rewording entered from the 782db6f side.

Authored commit 4733042bd ("Consolidate Decider recipe navigation")

Docs-only, no code smuggled in: removes the DUPLICATE mkdocs nav entry "Decider native text worker" (a leftover of the earlier additive conflict resolution) and consolidates recipe/README.md's two Decider bullets into one. Head has exactly one nav reference each to recipe/decider/README.md (line 75) and recipe/decider/validation.md (line 76); both files exist. No dead links introduced. mkdocs build --strict passes on the exact head.

Note (not a finding): intermediate state

First-parent intermediate 2c56521b is self-inconsistent: model.rs already said "only merged weights" while graph_loading.rs still asserted "require merged weights" — the profile-refusal test fails at that intermediate commit. 362ead0c (#113's b3cd06a2) fixes it. Head is consistent; bisectability note only.

CI (observed)

At the exact head 4733042bd, BOTH repos' Actions APIs show: rust=success, build=success, benchmarks=success, cuda-contract=success, deploy=skipped (intentional for PRs). ThinkFlowLab and InferMatrixLab runs 39886021650 / 39886021731. Green.

Host checks (commands run by me, miniconda python, no cargo/GPU)

  • git worktree add /tmp/pr-review-pass84/wt149 origin/pr/149 (removed after review).
  • python -m mkdocs build --strict → PASS (0.40s, zero warnings).
  • PYTHONPATH=. python -m pytest tests/benchmarks -q → 65 passed.

PR body / template

Template answers all present (Purpose, Dependency and review scope, Configuration/reproduction, Test plan and results, Demo/evidence, Required human contributor answers, Self-review). The body's top section correctly pins the current head: "Current head 4733042bd…, tree 579d40ad…" — I verified the tree hash equals git rev-parse origin/pr/149^{tree} exactly — and cites the current CI/Docs runs 39886021731/39886021650 as "success at this exact current head" (same runs the Actions APIs return at this head_sha). The evidence release link is re-pinned to pr113149-main-abi9-20261023.

Findings

Zero P0–P3 findings. (An initial P3 candidate — "body pins the old head" — was checked against the actual body and does not hold: the 428f834fa pin sits under the dated historical heading "## PR113 compatibility integration — 2026-10-22", and the body explicitly states "older sections below describe their explicitly pinned historical heads", with the top section carrying the current head, tree hash, and CI links. Dated, explicitly-pinned history is the intended structure; the historical section even honestly notes "this newly combined head needs fresh review; older approvals do not attest to it." Correctly classified as structure, not staleness.)

Informational notes (not findings): #109-attributed admission carry, with #109's known P3 (stale Reservation doc) actually resolved in this tree by the merge resolution — #109 itself still carries it; admission tests unioned with no loss; intermediate 2c56521b is self-inconsistent (test/message mismatch fails at that commit, fixed by the next merge — bisectability note only); main-side content mostly byte-identical to main; contract.md example 7→4 aligns with main.

Prior-finding status: the oversize→422 Low remains fixed (unchanged since cb5dd109); #109's P3 does not regress — it is resolved in this tree; no prior finding regresses.

Verdict

APPROVED — non-draft, zero findings, template answers present, merge resolutions verified (one manual resolution in admission.rs taking #109's canonical form, two pure auto-merges), entering content fully attributed (#109 code byte-identical to its approved head, #113's authored content byte-identical, main-side content reviewed), CI green at the exact head in both repos, mkdocs build --strict and 65 benchmark tests pass locally.

Material unverified checks (approval is not complete validation): A800 GPU/HTTP qualification remains author-reported with the named release archive (no GPU on this host; raw evidence audited by the contributor, not re-derived here); the Rust suite rests on the exact-head CI runs (green), not local execution; the intermediate-commit test inconsistency means a bisect through 2c56521b hits a failing test (head state is consistent).

@psv666

psv666 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Additional independent review at 4733042bd35263aab7e0c6ffabe9be0f5593b977: no actionable defect found in the inspected admission/deadline integration.

I traced request reservation before upload, complete-row token costs, permanent oversize 422 vs transient 503/Retry-After, cancellation while queued or in the blocking pool, retained capacity after HTTP timeout, and executor synchronization/retirement. The request-only unit-accounting fix and corrected reservation documentation are present in this branch.

Personally run successfully:

  • Runtime + Decider component tests: 72 including doctests passed / 9 ignored.
  • Full workspace fmt, strict all-target Clippy, 307 tests including doctests passed / 57 ignored, release build, and strict docs.
  • 22 Python verification-tool tests.
  • Two real CUDA selected-head tests, covering row order, cardinality and BF16 rounding.

GPU work used nonblocking reservations and a freshly compiled matching ABI9 library. I did not load the pinned trained Decider export or repeat the complete HTTP/model/fault-retirement campaign. CPU cancellation/deadline tests plus separate head tests are useful evidence, but do not constitute that end-to-end GPU qualification or a new performance claim. This comment does not submit approval.

Review provenance and scope
  • Head 4733042bd35263aab7e0c6ffabe9be0f5593b977; PR API base 5ba6a3ba668cb33028eb0bce2c6d17cdc53aebe7; diff merge-base 5ba6a3ba668cb33028eb0bce2c6d17cdc53aebe7. Head checked unchanged before publication. Tests used isolated source archives and per-PR Rust build directories.
  • Code/test diff +3805/−226; total text diff +4882/−252; 56 changed files. Code counts exclude generated JSON fixtures/docs.
  • Repository ID 1386454853. Loaded trusted .agents/skills/system1-omni-review/SKILL.md from target main@5ba6a3ba668cb33028eb0bce2c6d17cdc53aebe7, SHA-256 ae2dfac116376ae2ec41f65122cc1bf5aea10751b8b907198a56154e5d67230e, plus architecture, repository contracts, CONTRIBUTING, self-review artifact/demo rules, current workflows and affected component docs. Observed CI is recorded separately from personally run checks.

@hsliuustc0106

Copy link
Copy Markdown
Contributor

fix conflicts

Copy link
Copy Markdown
Collaborator Author

@hsliuustc0106 Conflicts with main are resolved and pushed; local checks pass. Could you please re-review? Thanks!

@hsliuustc0106

Copy link
Copy Markdown
Contributor

more conflicts

Copy link
Copy Markdown
Collaborator Author

@hsliuustc0106
hsliuustc0106 merged commit cd302b6 into InferMatrixLab:main Oct 24, 2026
6 checks passed
@hsliuustc0106 hsliuustc0106 mentioned this pull request Oct 24, 2026
4 tasks done
twu3202 added a commit to twu3202/system1-omni that referenced this pull request Oct 25, 2026
MrDongsls pushed a commit that referenced this pull request Oct 25, 2026
Brings in #99, #113, #127, #139 and #149. #139 added its own
image_decode.rs with decode_data_url; it keeps that entry point and its
limits, now on top of this branch's decode_rgb8, and Cua-S1's
image_request.rs takes main's version. The README and supported-models
rows keep this branch's OmniJev line and main's D3 line, and the Model
constructor keeps both vocab_buffers and main's graph_stats.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants