hsliuustc0106 · Oct 24, 2026 · Oct 19, 2026 · Oct 21, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -54,6 +54,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
 
 ### Changed
 
+- `Choice.confidence` / `Noul.confidence` now mean the same thing on every backend: `max_k p_k` for a choice and
+  `max(p, 1 - p)` for a noul, both computed in `validation.py` from the validated probabilities. Historical `conf=`
+  values from `laya` / `laya-served` (normalised entropy), `jev` / `clm` (service-reported) and `omnijev` (Jev margin)
+  are not comparable with new ones. Each backend's own number is kept as `backend_confidence` on the answer, in
+  `Decision.raw["backend_confidence"]`, and on tool/browser tick records when present (browser ticks also carry
+  `backend_confidences`, every answered head's number by question name). A backend's own `raw["backend_confidence"]`
+  is kept as `raw["backend_confidence_reply"]` instead of being overwritten. See #89.
+
 - Review guidance (`self-review`, `review-pr`) now treats merge readiness as a gate: authors confirm the pull
   request reports mergeable with CI green on the recorded head before requesting review, and reviewers do not
   approve a conflicting branch or a run pinned to an older commit.

diff --git a/docs/decision-models.md b/docs/decision-models.md
@@ -22,7 +22,13 @@ Input: an `Observation(state, images=())` (a JSON object or plain text) and type
 `ChoiceQuestion(options, goal=, rules=, operation=)` picks one key among `options` (a string per key, or a dict for
 the browser's element rows); `NoulQuestion(question, criteria=)` asks whether a statement holds. Output: a
 `Decision(answers, latency_ms, usage, model, raw)` with one `Choice(key, probabilities, confidence)` or
-`Noul(p, confidence)` per question name.
+`Noul(p, confidence)` per question name. **`confidence` always means the same thing on every backend:** for a
+choice it is `max_k p_k` (the peak probability); for a noul it is `max(p, 1 - p)`. Validation computes both from
+the validated probabilities in `s1a/decision_models/validation.py`, so Laya's normalised entropy, Jev's normalised
+margin, and OmniJev's recompute never become the reported `confidence`. Each backend's own number is kept as
+`backend_confidence` on the answer and under `Decision.raw["backend_confidence"]` (a map of question name → number)
+when the backend sent one in [0, 1]. These numbers are **not** calibrated unless a backend says so; do not reuse a
+threshold across backends.
 
 `Usage.known` distinguishes reported zero tokens from missing or malformed provider usage. A local backend can
 return `Usage()` for a known zero; `Usage.from_payload` preserves tolerant counts but marks incomplete usage unknown.
@@ -41,7 +47,8 @@ decision.choice("pick").key, decision.latency_ms, decision.usage.input_tokens
 `decide_many` is the one place validation happens. It refuses, with `MODEL_SERVICE_CONFIG_ERROR`, an empty
 question set, an image on a text-only model and a question type the backend lacks. It checks every answer:
 a choice must name an offered key, its probabilities must cover exactly the offered keys, lie in [0, 1], sum to 1
-within 0.02 and peak at the key; a noul probability lies in [0, 1] and its confidence defaults to `max(p, 1 - p)`.
+within 0.02 and peak at the key; a noul probability lies in [0, 1]. After those checks, `Choice.confidence` is set
+to `max_k p_k` and `Noul.confidence` to `max(p, 1 - p)` regardless of what the backend put in its `confidence` field.
 An unusable answer is re-asked with the same request up to `attempts` times (never on a deterministic backend, never
 after a transport error) and then raises `MODEL_CALL_FAILED`. `decide`, `choose` and `ask` are one-question
 shorthands; `warm()` and `close()` open and release the backend.
@@ -56,7 +63,7 @@ shorthands; `warm()` and `close()` open and release the backend.
 | `laya-served` | `ServedLayaModel(client, model=, max_len=)` | `laya-served` | one `POST /v1/systemone` per request to `LAYA_SERVED_URL`, body built with `laya_question()` as for `laya`; one retry on a dropped connection, 502 or 504, and after `Retry-After` on 503, all within `LAYA_SERVED_TIMEOUT_S`; the same window check as `laya`; `model` is the served checkpoint and revision, and `raw` keeps `served_by`, `url`, `request_id` and `server_timing` (see [served-laya.md](served-laya.md)) |
 | `cua` | `CuaS1Model(scorer, collator, model=, context_bytes=, option_bytes=)` | `cua` | Cua-S1 Nano, one `score_elements` pass per request on a thread; choice questions only, text only, deterministic; the context is header, state and rules; the checkpoint reads its first 256 bytes, and the first overflowing request logs one warning; `from_env` reads `CUA_S1_*` (see [configuration.md](configuration.md)) |
 | `omnijev` | `OmniJevModel(agent, model=)` | `omnijev` | OmniJev (a Qwen3.5 vision-language decision model), one `MSO1.system_one` call per request on a thread; choice and noul questions over the observation's screenshot, deterministic; the text state, goal and rules go in front of each question (a browser state puts its recent actions first and caps the page text at 2,000 characters, so a long page cannot push the history out of the 6,000-character context), a browser element row becomes its label and value; a choice's probabilities, which OmniJev leaves `abstain` out of, are renormalised over the offered keys; an observation without an image is `MODEL_CALL_FAILED`; `from_env` reads `OMNIJEV_*` (see [configuration.md](configuration.md)); in process only: system1-omni has no OmniJev vision worker yet (its supported models are Laya, Open-Jev text and Cua-S1 screenshots), so there is no served `omnijev` path |
-| `random` | `RandomModel(seed)` | `random` | uniform over the offered keys, confidence 0, one seeded stream per episode; choice questions only |
+| `random` | `RandomModel(seed)` | `random` | uniform over the offered keys (contract confidence = `1/|options|`), one seeded stream per episode; choice questions only |
 | `rule` | `RuleModel(name, rule)` | the rule's name | one-hot, confidence 1; a key outside the menu raises `RuntimeError`, a bug in the rule |
 
 `name` lands in every tick's `source` and in `Episode.policy`; the eval table's columns take their labels from it.
@@ -80,7 +87,7 @@ parameters) answer the same `choice` question in process.
   reads the validated decision onto a candidate. Ticks keep `decision_ms` and `input_tokens` for the artifact readers.
 - Rails (`DecisionModelRail`): one `check` question, noul or choice, banded by the spec's thresholds.
 - `decide`, `probe` and the MCP server: one `pick` question; the printed dict stays `choice`, `probabilities`,
-  `confidence`, `ms`.
+  `confidence` (= `max_k p_k`), `ms`, and `backend_confidence` when the backend sent its own number.
 
 Three oracle tests in `tests/test_decision_models_jev.py` pin the tool, rail and browser bodies to the pre-layer JSON.
 

diff --git a/s1a/browser/action_space.py b/s1a/browser/action_space.py
@@ -69,6 +69,9 @@ class Move:
     latency_ms: int
     usage: Usage
     model: str
+    backend_confidence: float | None = None
+    # Every answered head's backend-native confidence (operation, target, text value), by question name.
+    backend_confidences: dict[str, float] = field(default_factory=dict)
 
 
 def submittable(item: dict[str, Any]) -> bool:
@@ -242,4 +245,10 @@ def interpret(decision: Decision, space: ActionSpace) -> Move:
         latency_ms=decision.latency_ms,
         usage=decision.usage,
         model=decision.model,
+        backend_confidence=operation.backend_confidence,
+        backend_confidences={
+            name: answer.backend_confidence
+            for name, answer in decision.answers.items()
+            if answer.backend_confidence is not None
+        },
     )
diff --git a/s1a/browser/decision_model.py b/s1a/browser/decision_model.py
@@ -344,6 +344,16 @@ async def _decide_message(self, messages: Any) -> AssistantMessage:
                 "operation": move.operation,
                 "target": move.candidate.item.get("label") if move.candidate else None,
                 "confidence": round(move.confidence, 3),
+                **(
+                    {"backend_confidence": round(move.backend_confidence, 3)}
+                    if move.backend_confidence is not None
+                    else {}
+                ),
+                **(
+                    {"backend_confidences": {name: round(v, 3) for name, v in move.backend_confidences.items()}}
+                    if move.backend_confidences
+                    else {}
+                ),
                 "elements": len(space.elements),
                 "elapsed_ms": round((time.perf_counter() - run.started_at) * 1000),
                 "probabilities": probabilities,  # the replay's bars: the settled head's top keys

diff --git a/s1a/decision_models/__init__.py b/s1a/decision_models/__init__.py
@@ -29,7 +29,14 @@
     Reply,
     Usage,
 )
-from s1a.decision_models.validation import choice_faults, validate_answers, validate_choice, validate_noul
+from s1a.decision_models.validation import (
+    choice_confidence,
+    choice_faults,
+    noul_confidence,
+    validate_answers,
+    validate_choice,
+    validate_noul,
+)
 
 __all__ = [
     "DECISION_MODEL_NAMES",
@@ -57,7 +64,9 @@
     "RuleModel",
     "Usage",
     "build_model",
+    "choice_confidence",
     "choice_faults",
+    "noul_confidence",
     "jev_question",
     "validate_answers",
     "validate_choice",

diff --git a/s1a/decision_models/baselines.py b/s1a/decision_models/baselines.py
@@ -32,7 +32,7 @@ async def _decide(self, observation: Observation, questions: dict[str, Question]
         for name, question in questions.items():
             assert isinstance(question, ChoiceQuestion)
             ids = question.ids
-            # the entropy of a uniform pick is total, so its confidence is none
+            # backend sends 0; validation replaces Choice.confidence with max p (= 1/|ids| for uniform)
             answers[name] = _choice(self._rng.choice(ids), {key: 1.0 / len(ids) for key in ids}, 0.0)
         return Reply(answers=answers, latency_ms=0, model=self.model)
 

diff --git a/s1a/decision_models/types.py b/s1a/decision_models/types.py
@@ -94,30 +94,41 @@ def type(self) -> QuestionType:
 class Choice:
     """A validated choice answer: the key, a probability per key (sum 1 within 0.02, peak at the key), a confidence.
 
-    Only ``validation.validate_choice`` builds one from a backend's answer.
+    ``confidence`` is always ``max_k p_k`` (set by ``validation.validate_choice``). ``backend_confidence`` is the
+    number the backend itself reported, when it sent one in [0, 1]; it is a different scale for Laya (normalised
+    entropy) and Jev (normalised margin) and must not be compared across backends.
     """
 
     key: str
     probabilities: dict[str, float]
     confidence: float
+    backend_confidence: float | None = None
 
     def as_dict(self) -> Json:
-        """The CLI and MCP output shape."""
-        return {"choice": self.key, "probabilities": dict(self.probabilities), "confidence": self.confidence}
+        """The CLI and MCP output shape; includes ``backend_confidence`` only when the backend sent one."""
+        out: Json = {"choice": self.key, "probabilities": dict(self.probabilities), "confidence": self.confidence}
+        if self.backend_confidence is not None:
+            out["backend_confidence"] = self.backend_confidence
+        return out
 
 
 @dataclass(frozen=True)
 class Noul:
     """A validated noul answer: the probability the statement holds and a confidence.
 
-    Only ``validation.validate_noul`` builds one from a backend's answer; a missing confidence becomes max(p, 1-p).
+    ``confidence`` is always ``max(p, 1 - p)`` (set by ``validation.validate_noul``). ``backend_confidence`` keeps
+    the backend's own number when it sent one in [0, 1].
     """
 
     p: float
     confidence: float
+    backend_confidence: float | None = None
 
     def as_dict(self) -> Json:
-        return {"noul": self.p, "confidence": self.confidence}
+        out: Json = {"noul": self.p, "confidence": self.confidence}
+        if self.backend_confidence is not None:
+            out["backend_confidence"] = self.backend_confidence
+        return out
 
 
 Answer = Union[Choice, Noul]

diff --git a/s1a/decision_models/validation.py b/s1a/decision_models/validation.py
@@ -2,6 +2,9 @@
 """The invariants every answer must satisfy, checked on the backend's dicts; a breach is ``MODEL_CALL_FAILED``.
 
 ``validate_choice`` and ``validate_noul`` are the only constructors of ``Choice`` and ``Noul`` from a backend's answer.
+``Choice.confidence`` is always ``max_k p_k`` and ``Noul.confidence`` is always ``max(p, 1 - p)``, computed here from
+the validated probabilities so every backend reports the same scale. A backend's own number is kept as
+``backend_confidence`` on the answer and under ``Decision.raw["backend_confidence"]``.
 """
 
 from __future__ import annotations
@@ -29,18 +32,27 @@ def unit_interval(value: object) -> TypeGuard[float]:
     return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) and 0 <= value <= 1
 
 
+def choice_confidence(probabilities: dict[str, float]) -> float:
+    """The contract confidence for a choice: the peak probability (``max_k p_k``)."""
+    return float(max(probabilities.values()))
+
+
+def noul_confidence(p: float) -> float:
+    """The contract confidence for a noul: how far ``p`` sits from a coin flip (``max(p, 1 - p)``)."""
+    return max(p, 1.0 - p)
+
+
 def choice_faults(fields: Json, ids: list[str]) -> list[str]:
     """Why ``fields`` is not a choice answer over ``ids``: empty when it is one.
 
     ``choice`` is one of ``ids``; ``probabilities`` covers exactly ``ids`` with values in [0, 1] summing to 1 (within
-    0.02) and peaks at the chosen key (ties allowed); ``confidence`` lies in [0, 1].
+    0.02) and peaks at the chosen key (ties allowed). Backend ``confidence`` is not checked here: validation
+    recomputes it as ``max_k p_k``.
     """
     faults: list[str] = []
-    key, distribution, confidence = fields.get("choice"), fields.get("probabilities"), fields.get("confidence")
+    key, distribution = fields.get("choice"), fields.get("probabilities")
     if not isinstance(key, str) or key not in ids:
         faults.append(f"choice {key!r} is not one of {ids}")
-    if not unit_interval(confidence):
-        faults.append(f"confidence {confidence!r} is not in [0, 1]")
     if not isinstance(distribution, dict):
         return faults + ["probabilities is not an object"]
     if set(distribution) != set(ids):
@@ -55,26 +67,41 @@ def choice_faults(fields: Json, ids: list[str]) -> list[str]:
 
 
 def validate_choice(fields: Json, ids: list[str]) -> Choice:
-    """The typed choice with its numbers normalised, or ``MODEL_CALL_FAILED`` naming every fault."""
+    """The typed choice with contract confidence ``max_k p_k``, or ``MODEL_CALL_FAILED`` naming every fault.
+
+    When the backend sent a finite confidence in [0, 1], it is kept as ``backend_confidence`` (Laya's normalised
+    entropy, Jev's margin, OmniJev's recompute, and so on); it never becomes ``Choice.confidence``.
+    """
     faults = choice_faults(fields, ids)
     if faults:
         raise build_error(
             StatusCode.MODEL_CALL_FAILED, error_msg="decisions answer is not an offered choice: " + "; ".join(faults)
         )
+    probabilities = {str(key): float(weight) for key, weight in fields["probabilities"].items()}
+    native = fields.get("confidence")
     return Choice(
         key=str(fields["choice"]),
-        probabilities={str(key): float(weight) for key, weight in fields["probabilities"].items()},
-        confidence=float(fields["confidence"]),
+        probabilities=probabilities,
+        confidence=choice_confidence(probabilities),
+        backend_confidence=float(native) if unit_interval(native) else None,
     )
 
 
 def validate_noul(fields: Json) -> Noul:
-    """The typed noul with ``p`` in the unit interval and a confidence, ``max(p, 1 - p)`` when the backend gave none."""
-    p, confidence = fields.get("noul"), fields.get("confidence")
+    """The typed noul with contract confidence ``max(p, 1 - p)``.
+
+    A backend-supplied confidence in [0, 1] is kept as ``backend_confidence``; any other value is dropped. The
+    reported ``Noul.confidence`` is always ``max(p, 1 - p)``.
+    """
+    p, native = fields.get("noul"), fields.get("confidence")
     if not unit_interval(p):
         raise build_error(StatusCode.MODEL_CALL_FAILED, error_msg="decisions answer is not a noul probability")
     p = float(p)
-    return Noul(p=p, confidence=float(confidence) if unit_interval(confidence) else max(p, 1.0 - p))
+    return Noul(
+        p=p,
+        confidence=noul_confidence(p),
+        backend_confidence=float(native) if unit_interval(native) else None,
+    )
 
 
 def validate_answer(fields: Any, question: Question) -> Answer:
@@ -89,10 +116,24 @@ def validate_answer(fields: Any, question: Question) -> Answer:
 
 
 def validate_answers(reply: Reply, questions: dict[str, Question]) -> Decision:
-    """Every asked question answered and valid, as a ``Decision``; answers nobody asked for are dropped."""
+    """Every asked question answered and valid, as a ``Decision``; answers nobody asked for are dropped.
+
+    Per-question backend-native confidences are copied onto ``Decision.raw["backend_confidence"]`` (a map of
+    question name to number) so step records and probes can still read the original scale. If the backend's
+    own ``raw`` already carried a ``backend_confidence`` key, that value is kept as
+    ``raw["backend_confidence_reply"]`` rather than silently overwritten.
+    """
     answers: dict[str, Answer] = {}
     for name, question in questions.items():
         if name not in reply.answers:
             raise build_error(StatusCode.MODEL_CALL_FAILED, error_msg=f"no answer for question {name!r}")
         answers[name] = validate_answer(reply.answers[name], question)
-    return Decision(answers, reply.latency_ms, reply.usage, reply.model, reply.raw)
+    raw = dict(reply.raw)
+    native = {
+        name: answer.backend_confidence for name, answer in answers.items() if answer.backend_confidence is not None
+    }
+    if native:
+        if "backend_confidence" in raw:
+            raw["backend_confidence_reply"] = raw["backend_confidence"]
+        raw["backend_confidence"] = native
+    return Decision(answers, reply.latency_ms, reply.usage, reply.model, raw)
diff --git a/s1a/mcp_server.py b/s1a/mcp_server.py
@@ -128,7 +128,7 @@ async def decide(
     rules: str,
     model: Literal["jev", "clm", "laya", "laya-served", "cua"] = "jev",
 ) -> Decision:
-    """One choice question: the chosen key, probabilities, confidence and decision latency in ms.
+    """One choice question: the chosen key, probabilities, confidence (= max p) and decision latency in ms.
 
     Use jev over HTTP (default), laya/cua locally after installing the matching extra, or laya-served against a
     running Laya server (LAYA_SERVED_URL). Local model loading

diff --git a/s1a/probe.py b/s1a/probe.py
@@ -72,7 +72,7 @@ async def pick(
 ) -> dict[str, Any]:
     """One ``choice`` question over ``options``: the chosen key, a probability per option, the confidence, the latency.
 
-    The dict the CLI prints and the MCP server returns: ``choice``, ``probabilities``, ``confidence``, ``ms``.
+    The dict the CLI prints and the MCP server returns: ``choice``, ``probabilities``, ``confidence`` (always ``max_k p_k``), ``ms``, and ``backend_confidence`` when the backend sent its own number.
     """
     decision = await decision_model.decide_many(Observation(state), {"pick": ChoiceQuestion(options, rules=rules)})
     return {**decision.choice("pick").as_dict(), "ms": decision.latency_ms}