[Feat] Use native JEV-VL for browser decisions by linear3735 · Pull Request #78 · ThinkFlowLab/system1-agents · GitHub
Skip to content

[Feat] Use native JEV-VL for browser decisions - #78

Draft
linear3735 wants to merge 7 commits into
ThinkFlowLab:mainfrom
linear3735:codex/jev-vl-served
Draft

linear3735 wants to merge 7 commits into
ThinkFlowLab:mainfrom
linear3735:codex/jev-vl-served

Conversation

@linear3735

@linear3735 linear3735 commented Oct 14, 2026 •

Copy link
Copy Markdown

Required human contributor answers

The four human answers are reused from the explicit contributor intent recorded in System1-Omni #139, whose stated outcome includes Agents integration. The contributor reviewed this complete Agents diff and explicitly authorized its submission.

That retained human review attestation applies to the original submitted adapter diff at 0def927; it is not a new attestation that the contributor personally reviewed this merge head. The present main integration and documentation follow-up were independently reviewed by Codex/Kimi and pushed under the contributor’s explicit authorization for this conflict-refresh task.

  1. What outcome are you trying to achieve? Validate online-vision correctness and complete-request latency, then integrate with Agents.
  2. What is the highest-leverage variable for that outcome? Inference speed, prioritizing single-batch performance while validating correctness.
  3. How will this work move that variable rather than create busywork? Reuse #117, compare all three official decision types, changed images and cache invalidation, then measure complete latency for fresh images and multiple questions on a retained image. Optimize measured hotspots after validation.
  4. Who is the customer or user, and what do they actually need? System1 community developers working on inference and evaluation, with an online-image path and reproducible correctness and latency measurements.

Is a PR needed? Yes. The contributor explicitly requested publishing this reviewed Agents integration after the Omni online-image change.

Observed failure evidence: This is a backend integration feature. Malformed PNG data and huge integer probabilities/usage were observed during local negative probes and are pinned by the adapter regressions. The previously missing backend registration is covered by registration tests. Probe outputs and independent review reports are retained outside Git and are not yet public artifacts.

Why

Browser agents cannot currently send their current screenshots to the native JEV-VL worker from Omni #139. This adds --model jev-vl-served with explicit JEV_VL_URL, so the existing browser policy can use that worker for operation selection.

How

browser Observation (current original PNG/JPEG + state)
  → ServedJevVLModel → native /v1/systemone
  → validated Choice/Noul answers → existing browser tools

The adapter enforces image/body limits and PNG integrity, preserves option indices, uses one absolute deadline including image preparation, and fails the complete decision on invalid data. It performs no HTTP retry, answer re-ask or partial action. A single-candidate choice is forced locally after image validation and makes no model request. Chat/value still supplies free text and summaries.

Review order: jev_vl_served.py and its contract tests; factory/browser registration; browser provenance; then the two shared integer-overflow checks in types.py/validation.py. Existing browser execution and tool loops are reused. No new dependency, tool/desktop/rail integration, or CLI decide registration is added.

What

  • Add the served adapter, browser/factory registration, request/response provenance and 117 new test cases; update existing model-name assertions.
  • Reject huge integers without an overflowing float conversion in the shared validation helpers.
  • Document configuration and the bounded H800/browser validation, with its failure history and forced-choice exclusions.

Base/merge base: 0fe20f5044d455126c81f668ac9d28d706e6d3c0; current head: 5f6f0d646f984c5f61c508c854242ddd97b95e05. Full diff: 16 files, +1,250/−14 (git diff --numstat 0fe20f5..5f6f0d6). The 11 JEV-specific Python/test blob identities against the frozen runtime version 0def92771746ec94adc38bf41be9720bccfbf19d held through 0dd1693; the 2026-10-21 review follow-up below intentionally changes three of them (jev_vl_served.py and its two test modules), so the executed runtime evidence below stays bound to 0def927. This statement is limited to those files: importing main also brings its separate MCP, public231, ticket/evaluation and other upstream changes.

Review follow-up (2026-10-21, head 5f6f0d6)

Single commit 5f6f0d6 resolves all three points of Birfy's 2026-10-19 re-review: (1) decision_raw is retained only for jev-vl-served browser ticks, so laya-served/clm keep their provenance shape without a full response payload (a regression test covers both existing backends); (2) raw per-response headers are restricted to an allowlist (server-timing, x-request-id, content-type, content-length) so proxied deployments cannot leak cookie/authorization material into evidence (the fixture now plants private headers to prove they are dropped); (3) forced single-option choices record a null per-response request_id, and an all-forced decision a null top-level request_id, because no HTTP request is sent. Local verification on this head: python -m pytest -q 1,109 passed / 117 skipped / 204 subtests passed; ruff check . and ruff format --check . clean. The PR stays Draft pending re-review and maintainer approval; no new GPU/browser run was performed for this follow-up.

The normal merge preserves both the upstream MCP and JEV-VL CHANGELOG entries. It adds the native JEV-VL row requested in docs/capabilities.md, binding the existing public evidence and its limits to Agents 0def927 / Omni 26dc76b. The follow-up documentation commit replaces the obsolete statement that all run-local media are unpublished: the same-trial video and sanitized browser R3 packet are public, while the complete GPU raw campaigns remain unpublished. No evidence archive or historical runtime result is changed.

Verification

Local checks below ran on merge candidate d53c9c985aa4d5c20047331e027f9609590b75c4, using the existing Python environment and this checkout on the import path. On final head 0dd1693, diff checks and independent Codex review verified that the follow-up changes only the publication paragraph; the R1 tree is otherwise byte-identical. The full CPU suite was not repeated for that documentation-only delta. They are CPU/fixture checks, separate from exact-head GitHub CI and the retained runtime evidence.

  • ruff format --check ., ruff check . and ty check: passed.
  • python -m pytest -q: 1,107 passed / 117 skipped / 204 subtests passed; scripts/smoke.sh: passed. The pinned-reference public231 benchmark check passed 82 tests. Skips are not passes; smoke covers CLI/MCP, not real inference.
  • CHANGELOG.md and docs match the implementation and bounded evidence.
  • uv pip check --python <existing-env-python> passed. The first python -m pip check attempt could not run because this uv-managed environment has no pip; that failure is retained in the local record.

Fresh independent Codex and Kimi R1 source/merge reviews passed at d53c9c9; both independently reviewed the final documentation-only follow-up at 0dd169315a0f98c5b7e380d50a8310269c691e5d and passed. R1 covers unchanged feature content and R2 covers the exact publication paragraph and its context. Codex independently ran targeted tests and checked Git/blob identities; Kimi reviewed frozen source and command records without executing tests. Contributor reviews are not maintainer approval. Independent earlier reviews of frozen 0def927 and Birfy’s offline verification of 7250a50 remain historical checks with their stated boundaries; they are not new-head GPU/browser trials.

As of 2026-10-21, GitHub head matches 5f6f0d646f984c5f61c508c854242ddd97b95e05 and is MERGEABLE (no conflicts; overall merge state BLOCKED, i.e. Draft with review required). Exact-head Actions: test run 39622924723: benchmarks, core (3.11/3.13) and full SUCCESS; browser and Windows jobs skipped by workflow conditions (they establish neither Windows nor live-browser coverage). The previously reported green run at 0dd1693 and the green Actions run at old head ecfa02f are historical. No PR is merged or made ready by this update.

Demo / evidence

Same-trial R3 evidence packet (prepared offline from the retained records; no GPU/model/browser rerun):

  • Recorded replay: https://github.com/linear3735/system1-agents/releases/download/jevvl-r3-419202-evidence/evidence-replay.mp4 — 294 frames / 11.76 s, original full application left and static same-trial context right; no cuts or speed change. SHA-256 58550ee293bbfb9c5b1f563c03e8cbdb6f6ac2a3ef23080f73ec3435b932a9e4.
  • Sanitized raw records: https://github.com/linear3735/system1-agents/releases/download/jevvl-r3-419202-evidence/jevvl-r3-evidence.zip — 1083479 bytes, SHA-256 8e3242a6c275c4c1528f8a7643d2b8dcac2db3e9e2ebef4a1195cc8a095d78c6; 211 retained source/support files plus README, separate original/published hash manifests, redaction notes, all three attempts and a standard-library offline verifier. Extract and run python3 verify_bundle.py; no key/network/GPU is needed.
  • Included same-R3 evidence: 7 unchanged native HTTP request/response body pairs, 4 original decision PNGs, all operation/forced-choice records, tool calls and POST/oracle, the exact runner recording adjustment, source/binary/loaded-library and service identity, both READY and canonical RELEASED lease snapshots. Cluster paths, owners, device UUIDs, infrastructure addresses and the auxiliary provider endpoint use consistent aliases; the private alias dictionary is not published. Native wire bodies, PNGs, raw WebM and replay MP4 retain their original bytes.
  • Preserve R1 transport failure, R2 task success with empty recording, and R3 complete evidence acceptance; oracle verified 2/3, complete evidence 1/3. Auxiliary-model checkpoint/cost, broad task quality and browser speedup remain unverified. The original 107 renderer source hashes were checked before copying. verify_bundle.py checks consistency of the published records; it does not rerun model quality or independently authenticate a redacted original.

Birfy rechecked the published packet on 2026-10-15 and confirmed the earlier reviewer-access gap was resolved, leaving the review as COMMENTED. The 2026-10-17 follow-up requested a conflict-free main integration, exact-head CI and the native validation row supplied by this update. Keep Draft pending formal review; no maintainer approval or PR merge is claimed.

  • Executed versions: Agents 0def92771746ec94adc38bf41be9720bccfbf19d, Omni 26dc76bf65c544815e7b285866791da1e703816d; single H800, native Rust/CUDA JEV-VL serving, pinned autotrust/JEV-27B-VL@f34b598d4ef4bcefd337bee8d8e7ddd3b7733ccc.
  • The existing RECOVERABLE fixture was run through the production browser/live evaluator with recovery off, repeat 1, 120 s / 18 steps / stall 3. R3 used a run-local evaluator wrapper to close the owned browser after its final screenshot and finalize the recording; that wrapper is not product code in this PR.
  • R3: CLICK → TYPE_TEXT → CLICK → DONE, three executed actions, independent hello world POST verified, 12.451 s. Four non-forced operation choices; six forced targets are excluded from learned decisions; seven real visual HTTP requests returned 200 with the original screenshots. Text/summary came from the separately configured chat/value service; its provider alias was recorded, but its actual backend checkpoint and complete cost were not independently established.
  • Preserve all attempts: R1 transport timeout; R2 task oracle passed but video was empty; R3 complete evidence passed. Oracle verified 2/3 attempts, complete evidence 1/3; these are not statistical quality estimates.
  • Same-trial evidence replay: 294 frames / 11.76 s, no cuts or speed change, original application left and static same-run evidence panel right. No exact video-frame/client-UTC mapping or per-request server log is claimed. SHA-256: 58550ee293bbfb9c5b1f563c03e8cbdb6f6ac2a3ef23080f73ec3435b932a9e4.
  • Separate serving evidence: correctness 612 requests, eviction 36 requests, and 60 paired HTTP episodes / 1,080 timed requests. The full-request HTTP comparison is not an end-to-end browser speedup; some cold score/JPEG P95 results were slower. The published packet covers browser R3 and its service support records, not the complete raw 612-request correctness / 36-request eviction / 1,080-request timing campaigns; those full GPU raw archives remain unpublished. Run ID and scope are in the validation notes.

Same-trial evidence links are supplied above for independent rechecking. This PR remains Draft pending formal review. Exact-head Actions status is reported in Verification above; formal maintainer approval remains required before merge. This single-fixture smoke does not establish broad task quality, Flights completion, every frontend, or production acceptance. Omni #139 and its #117 shared backend remain separate upstream dependencies; no PR is merged by this submission.

linear3735 added 3 commits October 14, 2026 01:06
Replace four stale unverified statements with the single-H800 and RECOVERABLE R3 scope. Preserve forced-choice exclusions, failed attempts and the external-evidence publication gap. Runtime, tests and configuration remain at validated 0def927.

Complete candidate: 16 files, 1207 changed lines; authored Python and tests: 1143 changed lines.

Birfy commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Preserve the upstream Public231 entry and the unchanged JEV-VL entry under Added. Move JEV-VL below the existing Cua entry so the current main merges cleanly without rewriting either entry. No runtime, test or configuration changes.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at ecfa02f against main f91d006.

The only change since my verification of 7250a50 is to CHANGELOG.md (+6/−3). It moves the Cua-S1 4B entry above jev-vl-served and keeps the Public231 entry from main word for word. I checked that by merging locally: git merge-tree --write-tree origin/main ecfa02f exits 0. The merged ### Added list has Public231, then Cua-S1 4B, then jev-vl-served, then desktop text input, then Snake. Nothing is duplicated or dropped. The blank line after the Public231 bullet comes from main, not from this PR. Runtime code and tests are unchanged from the head I reran, and exact-head CI is green: core 3.11 and 3.13, full, and benchmarks. Windows and browser are skipped under the PR configuration.

I found no new code defect. The one remaining blocker is the one the PR already states: the same-trial video and the sanitized raw traces have no URL a reviewer can open. Keeping it in draft until they're linked makes sense. Once they are, I'm ready to re-check and approve.

@linear3735

Copy link
Copy Markdown
Author

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the published evidence for unchanged head ecfa02fa903ea4a3eab7f787babeaf7b81935abc.

Downloaded the ZIP without GitHub authentication; its SHA-256 matches 8e3242a6c275c4c1528f8a7643d2b8dcac2db3e9e2ebef4a1195cc8a095d78c6. Independent offline checks verified all 216 bundle-manifest files, 211 source records (107 renderer sources), seven HTTP-200 request/response body hashes, byte-identical decision PNG links, and response argmax choices. The records preserve CLICK → TYPE_TEXT → CLICK → DONE, six forced-target exclusions, the hello world POST/oracle, all three attempts, and the RELEASED lease.

The embedded replay matches the advertised MP4 hash; ffprobe reports 294 frames / 11.76 s. Sampled frames show locked → editing → submitted. The right-hand panel is static trial context, not synchronized server output.

This resolves the previously reported missing reviewer-accessible artifact gap. No new actionable defect found in this evidence check. These are consistency checks of retained, partly redacted records bound to Agents 0def927 / Omni 26dc76b, not independent authentication of the originals or a fresh model/browser/GPU run. Auxiliary-model checkpoint/cost, broad task quality and end-to-end speedup remain unverified; PR Windows/browser CI remains skipped. Formal maintainer acceptance is still separate. Leaving this as COMMENTED; Draft state unchanged.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review (Birfy)

Re-checked 2026-10-17: still draft and CONFLICTING with main. Prior feedback stands.

Please rebase onto current main (CHANGELOG / decision-model factory churn from recent merges and open stacks like #86) and re-request review once green and conflict-free. Keeping as draft until the native JEV-VL path has a clear validation row in docs/capabilities.md is still appropriate.

linear3735 added 2 commits October 18, 2026 00:44
Signed-off-by: linear3735 <heshming@qq.com>
Signed-off-by: linear3735 <heshming@qq.com>

@Birfy Birfy left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review (Birfy), head 0dd1693 — 2026-10-19

Thanks for the rebase and the evidence write-up. The branch is now conflict-free with main 0fe20f5, CI is green, and the full suite passes locally on this head (1107 passed, 117 skipped). The validation row in docs/capabilities.md and the "Served JEV-VL" section now say clearly what was validated (one bounded R3 fixture on one H800, all three attempts retained, forced choices excluded) and what was not, which is what I asked for last time.

The adapter itself reads well: exactly one PNG/JPEG, Pillow verify() + load() before a forced choice can skip HTTP, a single shared deadline across questions, and no silent re-ask.

A few things I'd like settled before this leaves draft:

  1. decision_raw in every browser tick (s1a/browser/decision_model.py). record["decision_raw"] = decision.raw is gated on decision.provenance, which is also non-empty for the existing HTTP backends laya-served and clm (both set url/request_id/served_by), not just JEV-VL. So this silently changes step records for the existing backends too: laya-served ticks will now carry its whole response payload. Either scope it to JEV-VL, or call it out in the CHANGELOG and add a test for the existing served backends' record shape.
  2. Full response headers are recorded. _post returns dict(response.headers) and it lands in raw["responses"][i]["headers"], then in decision_raw on disk. Behind a proxy or gateway that can include set-cookie or auth-related headers. Please keep an allowlist (server-timing, x-request-id, content type/length) rather than everything.
  3. Forced choices get a request_id for a request that was never sent. For a one-option choice the record carries a fresh uuid4 request_id with forced: true. Minor, but it makes it look like there's a server-side request to correlate; None would be clearer.

None of these block the design. Happy to approve once (1) and (2) are addressed and the PR is marked ready.

Edited: corrected point 1 (plain jev sets no provenance, so it is unaffected) and dropped a note about MAX_BODY_BYTES, which is enforced in _decide.

…ice IDs

Addresses the three points of Birfy's 2026-10-19 re-review at 0dd1693:
decision_raw is retained only for jev-vl-served ticks so the existing served
backends keep their provenance shape without a full payload; per-response
headers are restricted to server-timing, x-request-id, content-type and
content-length so proxied deployments cannot leak cookies or tokens into
evidence; forced choices carry a null request_id (and an all-forced decision a
null top-level request_id) because no HTTP request was sent.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants