Read the kind-word binding reply in the shapes the model actually writes so a whole base no longer stays untyped in silence by WaylandYang · Pull Request #894 · deeplethe/utopia · GitHub
Skip to content

Read the kind-word binding reply in the shapes the model actually writes so a whole base no longer stays untyped in silence - #894

Merged
WaylandYang merged 2 commits into
devfrom
claude/optimistic-thompson-0d420e
Sep 24, 2026
Merged

WaylandYang merged 2 commits into
devfrom
claude/optimistic-thompson-0d420e

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

What happened

Two of twelve identity-bench rounds ended with every document done, 30+ entities, 50+ facts, and every entities.type_id NULL, so all 21 anchors (located by class) were missing. The server log had no WARN.

The surviving KB (guard-run7 reverse) shows 13 align_types jobs, all done, no last_error, each taking 10–50 s, and zero type_bindings rows. Reproducing with the same entities against SiliconFlow deepseek-ai/DeepSeek-V3.2 showed why: at temperature 0 the model often answers the kind-word prompt not as the documented {"b": [[id, key]]} but as an id-keyed object, in several variants:

{ "0": null, "1": "organization", ... }
{ "0": ["organization"], "4": [null] }
{ "0": ["organization", "机构 — 研究院、研究所、集团…"] }
{ "0": [0, "organization"] }

parse_kind_word_response read any object without a b array as "no pairs, nothing malformed", so every item counted as unanswered. One unanswered vote out of two means no decision, nothing is written, nothing is logged, and nothing is re-enqueued (only call/parse failures were). The same prompt returns the same shape every round, so the base stayed untyped through all twelve documents. The "good" sibling round only bound anything on 2 of its 12 runs, for the same reason.

What changed

  • utopia-extract/align.rs: the parser reads all the shapes above, plus b as an object, {"id":..,"key":..} pairs and a bare top-level array. Unreadable ids/keys count as malformed instead of absent.
  • type_alignment.rs: a pass that parses to zero choices is treated as a failed pass and logged at WARN with finish_reason, char count and a snippet of the reply. The completion line reports unanswered, and a WARN follows when it is non-zero. Unfinished runs (failed or unanswered) re-enqueue themselves with a reask counter in the payload, at most MAX_REASK (3) times with a growing delay; new words or stale bindings still re-enqueue as before.
  • main.rs: the job dispatcher passes reask through.
  • Tests: four parser tests for the shapes; an_id_keyed_reply_still_binds_the_kind_word reproduces the all-NULL case through the scripted model and asserts the class lands; an_unreadable_reply_is_asked_again_a_bounded_number_of_times asserts no decision, one reask=1 job, and no more once the budget is spent.

Resolution, adjudication and governance are untouched.

Verification

  • cargo test --workspace, cargo clippy --workspace --all-targets -D warnings, cargo fmt --check green locally against a dedicated database.
  • Copying the broken KB's 29 entities into a fresh base: old binary binds 0 words, this branch binds 18 (typed 21/29).
  • Full identity bench, one round each order, SiliconFlow DeepSeek-V3.2 + bge-m3:
order missing f1 typed entities
forward 2 0.741 22/28
reverse 2 0.640 20/29

Earlier rounds had 3–9 anchors missing. The f1 numbers are not directly comparable with those rounds: with more anchors usable, the pre-existing INST/LN/P2 splits now enter the score (pair-for-pair identical to guard-run7 forward, which also had 2 missing). Those splits are candidate-generation gaps on the identity side, out of scope here.

Follow-ups noted separately: the phrase-binding parser has the same silent path; scripts/e2e_type_drift.mjs is stale (documents.find on a {docs: [...]} response) independent of this change.

🤖 Generated with Claude Code

WaylandYang and others added 2 commits September 24, 2026 12:00
…tes so a whole base no longer stays untyped in silence

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit bd66d04 into dev Sep 24, 2026
@WaylandYang
WaylandYang deleted the claude/optimistic-thompson-0d420e branch September 24, 2026 14:59
WaylandYang added a commit that referenced this pull request Sep 26, 2026
…tch needed the repair (#919)

gemini-3.5-flash through an OpenAI-compatible gateway writes a raw newline inside a
quoted reason. serde refuses it ("control character (\u0000-\u001F) found while
parsing a string"), the adjudicate_entities job retries three times on the same
reply, and the whole batch is marked failed; on a 100-document Re-DocRED base 2903
resolution_reviews stayed pending.

utopia-extract: parse_adjudication_repairing tries serde first; on failure it
escapes bare control characters inside string literals (\n, \r, \t, \u00XX) and
parses again, returning whether the repair was needed. parse_adjudication keeps
its signature and output. Newlines between tokens are untouched; \" does not end
a string; already-escaped sequences stay as they are.

utopia-server: adjudication.rs and governance.rs use the repairing parser and log
a WARN with the kb (and run) id, the batch size and the first pair's names when
the fallback was taken, the way #894/#895 report a reshaped alignment reply.

Tests: a well-formed reply is not repaired and reads the same as before; a reply
with a raw newline and a raw tab inside "reason"/"why" strings parses with the
newline kept in the reason; the escaper only touches control characters inside
strings.

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant