Every model call uses the workspace's reasoning effort, and none is a value: alignment no longer thinks on its own by WaylandYang · Pull Request #1066 · deeplethe/utopia · GitHub
Skip to content

Every model call uses the workspace's reasoning effort, and none is a value: alignment no longer thinks on its own - #1066

Merged
WaylandYang merged 2 commits into
devfrom
feat/judgement-follows-the-setting
Oct 3, 2026
Merged

WaylandYang merged 2 commits into
devfrom
feat/judgement-follows-the-setting

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Why

Alignment (kind words and phrases), rule proposal and the ontology agent used a fixed low reasoning effort, whatever the workspace set. That was decided on 2026-09-24 from a gemini measurement. On deepseek-flash low thinks seven or eight thousand tokens a call: in the warm-base run (#1065) those steps were six tenths of the tokens per document, and four fifths of that was thinking.

What changes

  • One setting for every call. chat_client_thinking and JUDGEMENT_EFFORT are gone; alignment, rule proposal and the ontology agent use the workspace's reasoning effort, like extraction, errata and time interpretation already did.
  • none is a value. The settings API and the column's CHECK (migration 0105) accept it. DeepSeek ignores minimal; only none turns its thinking off. The ninth bench run had to drop the CHECK by hand for this.
  • The settings page offers none, and its hint says every call uses the setting and that the value that turns thinking off depends on the endpoint.

Behaviour change for existing workspaces: a workspace set to minimal or none no longer gets low on alignment. One that wants the old behaviour sets low, which then also applies to extraction.

Measured

20 Re-DocRED documents, deepseek-flash, --judge 200 --errata, fresh base each. Thinking off: this branch, workspace none, two groups. Alignment on low: the 2026-09-30 run (dev @ 798fb24, extraction none).

Alignment low Off, group 1 Off, group 2
Tokens per document 47k 22k 19k
of which the three alignment steps 38k 10k 7k
Minutes 28 6 4
Judged precision after errata 91.7% 83.3% 89.7%
Misworded 5.7% 15.4% 7.5%
Signatures bound 75 80 52
Same-sentence recall after errata 19.2% 14.3% 12.0%
  • Tokens more than halve and the run is five times faster. Phrase alignment's output goes from 323k tokens to 4k with the input unchanged.
  • Precision stays above 0044's 75.3% threshold in both groups. Misworded goes up, in the direction gemini showed, by much less (gemini: 4% to 31%).
  • The two groups differ more than usual: 80 against 52 signatures bound. Without thinking, the two votes on a signature agree less from run to run. The low side is one run, so its own spread is not measured.
  • Recall on 20 documents is noisy and half of the gap is at extraction (entity-pair recall 14.0% against 11.0% and 12.9%), which alignment does not touch.

So this trades some precision and stability for tokens and time, and the workspace chooses. Against 0044's token threshold, 19k to 22k is still more than three times over.

Checked

  • The settings API accepted none against a freshly migrated database, and the model answered with it set.
  • cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo test -p utopia-cli (schema version 95), -p utopia-llm; pnpm typecheck, pnpm test (228).
  • The bench README has the section with the full table.

Notes for landing

🤖 Generated with Claude Code

WaylandYang and others added 2 commits October 3, 2026 13:25
… value: alignment no longer thinks on its own

Alignment, rule proposal and the ontology agent used a fixed low, set on
2026-09-24 from a gemini measurement. On deepseek-flash low thinks seven or
eight thousand tokens a call: those three steps were six tenths of the
tokens per document and four fifths of that was thinking. They now use the
workspace's setting like extraction, and the setting accepts none
(migration 0105), the only value that turns thinking off on DeepSeek.

Measured on 20 Re-DocRED documents with deepseek-flash, thinking off in two
groups against the 2026-09-30 run with alignment on low: tokens per document
22k and 19k against 47k, 6 and 4 minutes against 28, judged precision 83.3%
and 89.7% against 91.7%. A workspace that wants the old behaviour sets low.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…s migrations 0104 and 0105

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit fce1f8e into dev Oct 3, 2026
6 checks passed
@WaylandYang
WaylandYang deleted the feat/judgement-follows-the-setting branch October 3, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant