[Fix] Unify decision confidence as max_k p_k (keep backend-native as backend_confidence) by Birfy · Pull Request #91 · ThinkFlowLab/system1-agents · GitHub
Skip to content

[Fix] Unify decision confidence as max_k p_k (keep backend-native as backend_confidence) - #91

Open
Birfy wants to merge 2 commits into
mainfrom
issue-89-unify-confidence
Open

Birfy wants to merge 2 commits into
mainfrom
issue-89-unify-confidence

Conversation

@Birfy

@Birfy Birfy commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

Answer every question before implementation and PR submission, including drafts.
Reuse explicit, still-applicable human answers with a traceable source reference;
ask the human for anything missing and wait. An agent must not invent answers.
Blank fields, placeholders, or an agent-written rationale do not satisfy this gate.
Share only information and source links appropriate for this PR's audience.

  1. What outcome are you trying to achieve?

    Answer: One documented meaning for Choice.confidence / Noul.confidence on every backend, so conf= columns, MCP decide, and any future confidence gate compare the same scale (issue [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89; Danyang authorized filling intent from issue context for scheduled coding runs).

  2. What is the highest-leverage variable for that outcome?

    Answer: Where confidence is assigned — in validation.py from validated probabilities, not passed through from each backend's native formula.

  3. How will this work move that variable rather than create busywork?

    Answer: Recompute as max_k p_k / max(p, 1-p) in the single validation path; retain each backend's number as backend_confidence so provenance is not lost. No change to which option is chosen.

  4. Who is the customer or user, and what do they actually need?

    Answer: Operators and eval authors reading run summaries / probe / MCP decide output; they need confidence comparable across laya, jev, omnijev, and cua without silent scale mismatches (arXiv:2610.02267).

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes — issue #89 proposes one PR with the contract change, provenance field, docs, CHANGELOG, and offline tests. Scheduled coding run implements directly per Danyang's Oct 13 guidance.

For a bug fix: link actual observed failure evidence.
Evidence: Not a live thresholding bug (nothing gates on confidence yet). Scale mismatch is documented in #89 and Fast Models, Slow Evidence (arXiv:2610.02267): Laya confidence differs from max_k p_k by ~0.299 on average.

Why

Every backend returned a confidence that meant something different (Laya: normalised entropy; Jev/OmniJev: normalised margin; Cua: max p). Anything that thresholds or compares conf= across backends would silently break. Closes #89.

How

  • validation.validate_choice / validate_noul always set contract confidence from probabilities (max_k p_k / max(p, 1-p)).
  • Backend-native values land on Choice/Noul.backend_confidence, Decision.raw["backend_confidence"], and tool/browser tick records when present.
  • Docs (docs/decision-models.md), CHANGELOG, DecisionModelContract tests, and unit tests updated.

Files to open first: s1a/decision_models/validation.py, s1a/decision_models/types.py, tests/decision_model_contract.py.

What

  • Contract: Choice.confidence = peak probability; Noul.confidence = max(p, 1-p).
  • Provenance: backend_confidence on answers, raw, and ticks.
  • Docs + CHANGELOG note that historical laya/jev/omnijev conf= values are not comparable with new ones.
  • No change to which option is chosen.

Verification

  • uv run ruff format --check / ruff check / ty check on touched decision-model files
  • pytest -q — 1015 passed, 121 skipped
  • scripts/smoke.sh — ok
  • CHANGELOG.md and docs/decision-models.md updated

Demo / evidence

  • Video link, or exemption/blocker with follow-up: N/A — contract/validation change with offline unit tests; no agent run or System1-Omni path changed.
  • Application/task → agent actions → independently checked result: Unit tests with fakes and recorded answer shapes (Laya-like entropy 0.4 vs peak 0.9; OmniJev Jev-margin kept as backend_confidence).
  • System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Branch issue-89-unify-confidence on ThinkFlowLab/system1-agents; no Omni worker change.

Closes #89

Every backend used to pass its own confidence through validation, so Laya's
normalised entropy, Jev's margin and Cua's max p were not comparable. Compute
contract confidence in validation.py from the validated probabilities; keep
each backend's number as backend_confidence on the answer, in Decision.raw,
and on tool/browser ticks. Closes #89.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (own PR — commenting, not approving)

Reviewed the full diff against #89. Looks correct and ready for a human approval once CI is green.

What works

  • Contract confidence is computed once in validation.validate_choice / validate_noul as max_k p_k / max(p, 1-p), so backends can no longer silently disagree on scale.
  • Backend-native values land on Choice/Noul.backend_confidence, Decision.raw["backend_confidence"], and tool/browser ticks when present — matches the provenance acceptance criterion.
  • choice_faults correctly stops requiring a backend confidence in [0, 1]; out-of-range / missing natives are dropped rather than faulted.
  • Docs + CHANGELOG call out that historical laya/jev/omnijev conf= values are not comparable with the new contract — important for anyone reading old runs.
  • Tests cover the Laya-like entropy≠peak case, OmniJev Jev-margin as backend_confidence, RandomModel uniform → contract 0.5, and the DecisionModelContract peak-probability invariant. No change to which option is chosen.

Nits (non-blocking)

  • Browser ticks only carry backend_confidence from the operation head; other heads remain under Decision.raw["backend_confidence"]. That matches the previous Move.confidence shape — fine, just worth knowing when reading replays.
  • validate_answers overwrites any pre-existing reply.raw["backend_confidence"] key. Unlikely today; if a backend ever puts that key in raw itself, rename or nest.

Cannot approve my own PR. Once CI is green on this head, this is merge-ready pending human approval.

Birfy commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

@Birfy

Birfy commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin)

1 participant