Repository navigation
[Feat] Use native JEV-VL for browser decisions - #78
linear3735 wants to merge 7 commits into
Conversation
Replace four stale unverified statements with the single-H800 and RECOVERABLE R3 scope. Preserve forced-choice exclusions, failed attempts and the external-evidence publication gap. Runtime, tests and configuration remain at validated 0def927. Complete candidate: 16 files, 1207 changed lines; authored Python and tests: 1143 changed lines.
Preserve the upstream Public231 entry and the unchanged JEV-VL entry under Added. Move JEV-VL below the existing Cua entry so the current main merges cleanly without rewriting either entry. No runtime, test or configuration changes.
Birfy
left a comment
There was a problem hiding this comment.
Re-review at ecfa02f against main f91d006.
The only change since my verification of 7250a50 is to CHANGELOG.md (+6/−3). It moves the Cua-S1 4B entry above jev-vl-served and keeps the Public231 entry from main word for word. I checked that by merging locally: git merge-tree --write-tree origin/main ecfa02f exits 0. The merged ### Added list has Public231, then Cua-S1 4B, then jev-vl-served, then desktop text input, then Snake. Nothing is duplicated or dropped. The blank line after the Public231 bullet comes from main, not from this PR. Runtime code and tests are unchanged from the head I reran, and exact-head CI is green: core 3.11 and 3.13, full, and benchmarks. Windows and browser are skipped under the PR configuration.
I found no new code defect. The one remaining blocker is the one the PR already states: the same-trial video and the sanitized raw traces have no URL a reviewer can open. Keeping it in draft until they're linked makes sense. Once they are, I'm ready to re-check and approve.
Birfy
left a comment
There was a problem hiding this comment.
Rechecked the published evidence for unchanged head ecfa02fa903ea4a3eab7f787babeaf7b81935abc.
Downloaded the ZIP without GitHub authentication; its SHA-256 matches 8e3242a6c275c4c1528f8a7643d2b8dcac2db3e9e2ebef4a1195cc8a095d78c6. Independent offline checks verified all 216 bundle-manifest files, 211 source records (107 renderer sources), seven HTTP-200 request/response body hashes, byte-identical decision PNG links, and response argmax choices. The records preserve CLICK → TYPE_TEXT → CLICK → DONE, six forced-target exclusions, the hello world POST/oracle, all three attempts, and the RELEASED lease.
The embedded replay matches the advertised MP4 hash; ffprobe reports 294 frames / 11.76 s. Sampled frames show locked → editing → submitted. The right-hand panel is static trial context, not synchronized server output.
This resolves the previously reported missing reviewer-accessible artifact gap. No new actionable defect found in this evidence check. These are consistency checks of retained, partly redacted records bound to Agents 0def927 / Omni 26dc76b, not independent authentication of the originals or a fresh model/browser/GPU run. Auxiliary-model checkpoint/cost, broad task quality and end-to-end speedup remain unverified; PR Windows/browser CI remains skipped. Formal maintainer acceptance is still separate. Leaving this as COMMENTED; Draft state unchanged.
Birfy
left a comment
There was a problem hiding this comment.
Follow-up review (Birfy)
Re-checked 2026-10-17: still draft and CONFLICTING with main. Prior feedback stands.
Please rebase onto current main (CHANGELOG / decision-model factory churn from recent merges and open stacks like #86) and re-request review once green and conflict-free. Keeping as draft until the native JEV-VL path has a clear validation row in docs/capabilities.md is still appropriate.
Signed-off-by: linear3735 <heshming@qq.com>
Signed-off-by: linear3735 <heshming@qq.com>
There was a problem hiding this comment.
Re-review (Birfy), head 0dd1693 — 2026-10-19
Thanks for the rebase and the evidence write-up. The branch is now conflict-free with main 0fe20f5, CI is green, and the full suite passes locally on this head (1107 passed, 117 skipped). The validation row in docs/capabilities.md and the "Served JEV-VL" section now say clearly what was validated (one bounded R3 fixture on one H800, all three attempts retained, forced choices excluded) and what was not, which is what I asked for last time.
The adapter itself reads well: exactly one PNG/JPEG, Pillow verify() + load() before a forced choice can skip HTTP, a single shared deadline across questions, and no silent re-ask.
A few things I'd like settled before this leaves draft:
decision_rawin every browser tick (s1a/browser/decision_model.py).record["decision_raw"] = decision.rawis gated ondecision.provenance, which is also non-empty for the existing HTTP backendslaya-servedandclm(both seturl/request_id/served_by), not just JEV-VL. So this silently changes step records for the existing backends too:laya-servedticks will now carry its whole response payload. Either scope it to JEV-VL, or call it out in the CHANGELOG and add a test for the existing served backends' record shape.- Full response headers are recorded.
_postreturnsdict(response.headers)and it lands inraw["responses"][i]["headers"], then indecision_rawon disk. Behind a proxy or gateway that can includeset-cookieor auth-related headers. Please keep an allowlist (server-timing,x-request-id, content type/length) rather than everything. - Forced choices get a
request_idfor a request that was never sent. For a one-option choice the record carries a freshuuid4request_idwithforced: true. Minor, but it makes it look like there's a server-side request to correlate;Nonewould be clearer.
None of these block the design. Happy to approve once (1) and (2) are addressed and the PR is marked ready.
Edited: corrected point 1 (plain jev sets no provenance, so it is unaffected) and dropped a note about MAX_BODY_BYTES, which is enforced in _decide.
…ice IDs Addresses the three points of Birfy's 2026-10-19 re-review at 0dd1693: decision_raw is retained only for jev-vl-served ticks so the existing served backends keep their provenance shape without a full payload; per-response headers are restricted to server-timing, x-request-id, content-type and content-length so proxied deployments cannot leak cookies or tokens into evidence; forced choices carry a null request_id (and an all-forced decision a null top-level request_id) because no HTTP request was sent. Co-Authored-By: Claude Code <noreply@anthropic.com>

Required human contributor answers
The four human answers are reused from the explicit contributor intent recorded in System1-Omni #139, whose stated outcome includes Agents integration. The contributor reviewed this complete Agents diff and explicitly authorized its submission.
That retained human review attestation applies to the original submitted adapter diff at
0def927; it is not a new attestation that the contributor personally reviewed this merge head. The present main integration and documentation follow-up were independently reviewed by Codex/Kimi and pushed under the contributor’s explicit authorization for this conflict-refresh task.Is a PR needed? Yes. The contributor explicitly requested publishing this reviewed Agents integration after the Omni online-image change.
Observed failure evidence: This is a backend integration feature. Malformed PNG data and huge integer probabilities/usage were observed during local negative probes and are pinned by the adapter regressions. The previously missing backend registration is covered by registration tests. Probe outputs and independent review reports are retained outside Git and are not yet public artifacts.
Why
Browser agents cannot currently send their current screenshots to the native JEV-VL worker from Omni #139. This adds
--model jev-vl-servedwith explicitJEV_VL_URL, so the existing browser policy can use that worker for operation selection.How
The adapter enforces image/body limits and PNG integrity, preserves option indices, uses one absolute deadline including image preparation, and fails the complete decision on invalid data. It performs no HTTP retry, answer re-ask or partial action. A single-candidate choice is forced locally after image validation and makes no model request. Chat/value still supplies free text and summaries.
Review order:
jev_vl_served.pyand its contract tests; factory/browser registration; browser provenance; then the two shared integer-overflow checks intypes.py/validation.py. Existing browser execution and tool loops are reused. No new dependency, tool/desktop/rail integration, or CLI decide registration is added.What
Base/merge base:
0fe20f5044d455126c81f668ac9d28d706e6d3c0; current head:5f6f0d646f984c5f61c508c854242ddd97b95e05. Full diff: 16 files, +1,250/−14 (git diff --numstat 0fe20f5..5f6f0d6). The 11 JEV-specific Python/test blob identities against the frozen runtime version0def92771746ec94adc38bf41be9720bccfbf19dheld through0dd1693; the 2026-10-21 review follow-up below intentionally changes three of them (jev_vl_served.pyand its two test modules), so the executed runtime evidence below stays bound to0def927. This statement is limited to those files: importing main also brings its separate MCP, public231, ticket/evaluation and other upstream changes.Review follow-up (2026-10-21, head
5f6f0d6)Single commit
5f6f0d6resolves all three points of Birfy's 2026-10-19 re-review: (1)decision_rawis retained only forjev-vl-servedbrowser ticks, solaya-served/clmkeep their provenance shape without a full response payload (a regression test covers both existing backends); (2) raw per-response headers are restricted to an allowlist (server-timing,x-request-id,content-type,content-length) so proxied deployments cannot leak cookie/authorization material into evidence (the fixture now plants private headers to prove they are dropped); (3) forced single-option choices record a null per-responserequest_id, and an all-forced decision a null top-levelrequest_id, because no HTTP request is sent. Local verification on this head:python -m pytest -q1,109 passed / 117 skipped / 204 subtests passed;ruff check .andruff format --check .clean. The PR stays Draft pending re-review and maintainer approval; no new GPU/browser run was performed for this follow-up.The normal merge preserves both the upstream MCP and JEV-VL CHANGELOG entries. It adds the native JEV-VL row requested in
docs/capabilities.md, binding the existing public evidence and its limits to Agents0def927/ Omni26dc76b. The follow-up documentation commit replaces the obsolete statement that all run-local media are unpublished: the same-trial video and sanitized browser R3 packet are public, while the complete GPU raw campaigns remain unpublished. No evidence archive or historical runtime result is changed.Verification
Local checks below ran on merge candidate
d53c9c985aa4d5c20047331e027f9609590b75c4, using the existing Python environment and this checkout on the import path. On final head0dd1693, diff checks and independent Codex review verified that the follow-up changes only the publication paragraph; the R1 tree is otherwise byte-identical. The full CPU suite was not repeated for that documentation-only delta. They are CPU/fixture checks, separate from exact-head GitHub CI and the retained runtime evidence.ruff format --check .,ruff check .andty check: passed.python -m pytest -q: 1,107 passed / 117 skipped / 204 subtests passed;scripts/smoke.sh: passed. The pinned-reference public231 benchmark check passed 82 tests. Skips are not passes; smoke covers CLI/MCP, not real inference.CHANGELOG.mdand docs match the implementation and bounded evidence.uv pip check --python <existing-env-python>passed. The firstpython -m pip checkattempt could not run because this uv-managed environment has no pip; that failure is retained in the local record.Fresh independent Codex and Kimi R1 source/merge reviews passed at
d53c9c9; both independently reviewed the final documentation-only follow-up at0dd169315a0f98c5b7e380d50a8310269c691e5dand passed. R1 covers unchanged feature content and R2 covers the exact publication paragraph and its context. Codex independently ran targeted tests and checked Git/blob identities; Kimi reviewed frozen source and command records without executing tests. Contributor reviews are not maintainer approval. Independent earlier reviews of frozen0def927and Birfy’s offline verification of7250a50remain historical checks with their stated boundaries; they are not new-head GPU/browser trials.As of 2026-10-21, GitHub head matches
5f6f0d646f984c5f61c508c854242ddd97b95e05and is MERGEABLE (no conflicts; overall merge stateBLOCKED, i.e. Draft with review required). Exact-head Actions: test run 39622924723: benchmarks, core (3.11/3.13) and full SUCCESS; browser and Windows jobs skipped by workflow conditions (they establish neither Windows nor live-browser coverage). The previously reported green run at0dd1693and the green Actions run at old headecfa02fare historical. No PR is merged or made ready by this update.Demo / evidence
Same-trial R3 evidence packet (prepared offline from the retained records; no GPU/model/browser rerun):
58550ee293bbfb9c5b1f563c03e8cbdb6f6ac2a3ef23080f73ec3435b932a9e4.8e3242a6c275c4c1528f8a7643d2b8dcac2db3e9e2ebef4a1195cc8a095d78c6; 211 retained source/support files plus README, separate original/published hash manifests, redaction notes, all three attempts and a standard-library offline verifier. Extract and runpython3 verify_bundle.py; no key/network/GPU is needed.verify_bundle.pychecks consistency of the published records; it does not rerun model quality or independently authenticate a redacted original.Birfy rechecked the published packet on 2026-10-15 and confirmed the earlier reviewer-access gap was resolved, leaving the review as COMMENTED. The 2026-10-17 follow-up requested a conflict-free main integration, exact-head CI and the native validation row supplied by this update. Keep Draft pending formal review; no maintainer approval or PR merge is claimed.
0def92771746ec94adc38bf41be9720bccfbf19d, Omni26dc76bf65c544815e7b285866791da1e703816d; single H800, native Rust/CUDA JEV-VL serving, pinnedautotrust/JEV-27B-VL@f34b598d4ef4bcefd337bee8d8e7ddd3b7733ccc.CLICK → TYPE_TEXT → CLICK → DONE, three executed actions, independenthello worldPOST verified, 12.451 s. Four non-forced operation choices; six forced targets are excluded from learned decisions; seven real visual HTTP requests returned 200 with the original screenshots. Text/summary came from the separately configured chat/value service; its provider alias was recorded, but its actual backend checkpoint and complete cost were not independently established.58550ee293bbfb9c5b1f563c03e8cbdb6f6ac2a3ef23080f73ec3435b932a9e4.Same-trial evidence links are supplied above for independent rechecking. This PR remains Draft pending formal review. Exact-head Actions status is reported in Verification above; formal maintainer approval remains required before merge. This single-fixture smoke does not establish broad task quality, Flights completion, every frontend, or production acceptance. Omni #139 and its #117 shared backend remain separate upstream dependencies; no PR is merged by this submission.