Prefer grounded subject evidence before namesake profile scores by virtue039 · Pull Request #1251 · deeplethe/utopia · GitHub
Skip to content

Prefer grounded subject evidence before namesake profile scores - #1251

Draft
virtue039 wants to merge 2 commits into
deeplethe:devfrom
virtue039:codex/mention-local-namesake-evidence
Draft

virtue039 wants to merge 2 commits into
deeplethe:devfrom
virtue039:codex/mention-local-namesake-evidence

Conversation

@virtue039

@virtue039 virtue039 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Two exact namesakes can attach to the person with the stronger passage/profile similarity even when a source-grounded statement uniquely identifies the other person. Check the current subject's valid quotes before profile tiers, while retaining the existing fallback and review lifecycle.

Current head: 6b63f5919b40274d59bc61e4a0e242f57187e7ef, including latest dev f2e2a47a2b6d0508aab88dc1b3689a6fee44c606 (rechecked 2026-10-21). Merge conflict resolved; remains draft because incomplete asynchronous counterfactuals still limit regression attribution. This PR does not claim a stable overall accuracy gain.

Change

  • Pass only source-located, single-sentence quotes owned by the current mention's subject. Missing, ungrounded, cross-sentence and shared/overlapping evidence cannot trigger a new identity choice. A department elsewhere in a paragraph is not this person's evidence.
  • Across exact namesakes, a uniquely corroborating candidate wins before profile tiers. Supplement typed/untyped exact candidates only for this comparison; retain incompatible-type and caller exclusions and do not classify the winner automatically.
  • With absent/shared/conflicting evidence, retain existing profile tiers, single-candidate behavior and tie reviews; do not retry a conflicting clue against a reduced candidate set.
  • Preserve dev's Markdown name normalization and lazy degree queries. Match both folded and stored formatted spellings, protected by a new regression control in both seed orders.

No API, schema, extraction prompt, adjudicator, unresolved-entity pool or merge gate changes. Three files, +397/-19 lines. #1192/#1199 remain closed without merging; this change does not revive them. The negative-example implementation in #1334 is independent.

Validation at this head

All 40 fixed checks pass, including the original eight unchanged inputs/gold and #1220's adjudicator-input controls. Current dev still reproduces the original two passage-over-profile failures. Forward/reverse controls cover correct ownership, one candidate, unrelated co-occurrence, quote ownership/grounding, shared/conflicting clues, typed/untyped and incompatible candidates, caller exclusions, absent-evidence fallback and repeated mentions without extra entities/reviews. The two known-failure entries are removed. Removing stored-spelling compatibility makes the new Markdown check fail in both orders; the final restored implementation passes.

fmt, workspace/all-targets clippy (-D warnings), full workspace tests (1,645 passed, 0 failed, 6 pre-existing ignored), workspace build, frozen pnpm install/build, 308 web tests and 15 scorer tests pass. DB/PDF require guards were enabled. CI 39535994426 passed all six jobs at the exact head. Separate baseline/candidate artifacts were rebuilt from the frozen commits; hashes are recorded. Dependency caches were reused; these were not empty-cache builds.

Fresh same-session live-model comparison

2026-10-21: unchanged repository identity corpus/scorer; qwen3.7-flash-2026-07-15, thinking disabled; pinned BGE-M3 5617a9f61b028005a4858fdac845db406aefb181, CPU float32 normalized CLS, 1024 dimensions, max 8192 tokens. Two rounds, each two independent forward/reverse runs per side: 8 new measured KBs per side, 16 total. Interleaved sides in one session, round barriers, one active KB per side, serial documents. Registration's empty default KB is excluded. No score-selected reruns.

All 192 documents completed, with zero provider failures, unfinished documents or bad jobs. All 1,023 responses reported the requested model version. The preflight's 17 tokens are excluded. Model request options match per call mode: streamed calls use temperature 0 and max_tokens 65536; non-streamed calls retain repository/provider defaults. Both disable thinking. Local embedding request/token fields are not mixed into chat token totals.

Metric dev baseline candidate
Mean F1 0.560875 0.580250
Round 1 mean F1 0.603250 0.617250
Round 2 mean F1 0.518500 0.543250
False-merge pairs, total 22 21
False-split pairs, total 49 52
Anchor coverage 128/168 (76.19%) 131/168 (77.98%)
Live entities, total 218 222
Pending human reviews, total 2 0
Chat calls 512 511
Prompt / completion tokens 661,142 / 36,027 640,625 / 35,436
Total chat tokens 697,169 676,061
Local embedding requests 399 409

F1 is computed on found anchors, so pair errors must be read with coverage. Both new rounds have slightly higher F1, but splits/entities increase and the historical frozen revision's measurement moved in the opposite direction. These small samples do not establish a stable overall gain. The recorded live named-entity decisions also do not demonstrate a direct benefit from the patch; the deterministic controls establish the targeted fix.

All 16 KBs: score, coverage, missing anchors and work

F/R = forward/reverse. Human counts sum the original scorer's pending human rows, not review groups.

Run Side Order F1 Merge Split Covered/21 Entities Human Calls Chat tokens Missing
r1-b1 baseline F 0.692 3 5 17 26 0 62 78672 d00-inst, d02-p1-lead, d06-p2-lead, d08-p2
r1-b1 candidate F 0.583 3 7 17 28 0 64 86544 d00-inst, d02-p1-lead, d06-p2-lead, d08-p2
r1-b1 baseline R 0.583 3 7 17 26 0 68 91929 d00-inst, d02-p1-lead, d06-p2-lead, d06-p2-depth
r1-b1 candidate R 0.667 0 5 15 30 0 68 86940 d00-inst, d02-p1-lead, d06-ln, d06-p2-lead, d07-zw-eng, d08-p2
r1-b2 baseline F 0.667 0 7 17 29 0 66 88269 d00-inst, d02-p1-lead, d06-p2-lead, d06-p2-depth
r1-b2 candidate F 0.636 3 5 17 28 0 62 78936 d00-inst, d02-p1-lead, d06-ln, d06-p2-lead
r1-b2 baseline R 0.471 3 6 16 27 0 63 82989 d00-inst, d02-p1-lead, d03-p1, d06-ln, d06-p2-lead
r1-b2 candidate R 0.583 3 7 17 27 0 65 88733 d00-inst, d02-p1-lead, d06-p2-lead, d08-p2
r2-b1 baseline F 0.533 3 4 13 29 1 64 95989 d00-inst, d02-p1-lead, d03-p1, d06-ln, d06-p2-lead, d06-p2-depth, d08-p2, d09-p1
r2-b1 candidate F 0.583 3 7 16 27 0 60 77880 d00-inst, d02-p1-lead, d06-p2-lead, d06-p2-depth, d08-p2
r2-b1 baseline R 0.583 3 7 16 27 0 65 89771 d00-inst, d02-p1-lead, d06-p2-lead, d06-p2-depth, d08-p2
r2-b1 candidate R 0.526 3 6 16 27 0 65 86376 d00-inst, d02-p1-lead, d06-ln, d06-p2-lead, d08-p2
r2-b2 baseline F 0.583 3 7 17 27 0 54 76490 d00-inst, d02-p1-lead, d06-p2-lead, d06-p2-depth
r2-b2 candidate F 0.526 3 6 16 27 0 64 87529 d00-inst, d02-p1-lead, d06-ln, d06-p2-lead, d06-p2-depth
r2-b2 baseline R 0.375 4 6 15 27 1 70 93060 d00-inst, d02-p1-lead, d03-p1, d06-ln, d06-p2-lead, d08-p2
r2-b2 candidate R 0.538 3 9 17 28 0 63 83123 d02-p1-lead, d06-p2-lead, d06-p2-depth, d08-p2

Regression attribution

For the older frozen 70bf7a7 / c019e4c measurement, the previous full-tape controls reproduced 22/27 candidate-extra entries on the old baseline. Follow-up now reproduces three more at their first-error document boundaries with identical recorded requests/replies and all score fields. The last two resolver transitions also return identical choices, review proposals and original scores on restored pre-decision state (including original name-vector ownership); these isolate the resolver and do not pretend the two full asynchronous tapes completed. An initial restoration with incorrectly repointed name vectors is retained as an invalid attempt, not final evidence.

For the fresh f2e2a47 / 6b63f59 measurement, inspect all 25 candidate-extra entries: 4 merges, 13 splits, 8 missing anchors. Ten pair entries involve an anchor absent on the paired baseline and are coverage exposure, not directly comparable new pair errors. Source replies expose omitted subjects, conflated same-name subjects, described-person flags, alias declarations and unclassified English equipment. A reverse d04 merge is located in the subsequent adjudicator investigation choosing same (0.95). These observations do not classify everything as model noise.

Atomic pre-request identity snapshots reproduce all 16 final stock-scorer results exactly and locate 176 settled document prefixes. All 96 candidate document named-entity phases, not a score-selected subset, are replayed on the same captured candidate state/reply/vectors using both commits: 0 differing choices, profiles, name facts, review candidates or flags. The actual parser/store resolver and hashed caller-cache helpers participate. Downstream statement/name application, type alignment, adjudication and scheduling are outside that isolated control.

Additional complete-pipeline strict replays use the original 511 candidate replies and fixed embedding cache, map only dynamic UUIDs, and refuse unmatched inputs. They make zero provider calls and are kept separate from valid live measurements.

Complete diagnostic replays: 6; new entries reproduced on the baseline at the first-error boundary: 17/25. Incomplete replays remain incomplete; no mismatched input receives a substitute reply.

Candidate tape Side Matched / total Exit All score fields equal to primary Identity graph projection equal to primary
candidate-r1-b1/F baseline 30/64 1 incomplete incomplete
candidate-r1-b1/F candidate 30/64 1 incomplete incomplete
candidate-r1-b1/R baseline 68/68 0 True True
candidate-r1-b1/R candidate 68/68 0 True True
candidate-r1-b2/F baseline 62/62 0 True True
candidate-r1-b2/F candidate 62/62 0 True True
candidate-r1-b2/R baseline 43/65 1 incomplete incomplete
candidate-r1-b2/R candidate 43/65 1 incomplete incomplete
candidate-r2-b1/F baseline 60/60 0 True False
candidate-r2-b1/F candidate 60/60 0 True False
candidate-r2-b1/R baseline 42/65 1 incomplete incomplete
candidate-r2-b1/R candidate 42/65 1 incomplete incomplete
candidate-r2-b2/F baseline 29/64 1 incomplete incomplete
candidate-r2-b2/F candidate 29/64 1 incomplete incomplete
candidate-r2-b2/R baseline 40/63 1 incomplete incomplete
candidate-r2-b2/R candidate 40/63 1 incomplete incomplete

All five incomplete tape groups stop at the same first unmatched request on both commits (after UUID mapping). In each, the recorded investigation facts result contains both 负责 → 海探1 and 批复了 → 海探1, while the replay contains only the latter. This locates a tool-visible-fact reproducibility limit; it does not establish universal downstream equivalence. Nearest-text comparison is diagnostic only and never supplies a completion.

Eight entries remain beyond the validated strict replay prefixes; each still has an identical fixed-state named-phase control:

Run/order Entry First-error document
r1-b1/F d02-p1-depth ≠ d09-p1 (P1) d09-home-port.txt
r1-b2/R d02-p1-depth ≠ d03-p1 (P1) d02-first-dive.txt
r2-b2/F d06-ln d06-spec.txt
r2-b2/R d04-zw(ZW_FIN) = d07-zw-eng(ZW_ENG) d04-budget.txt
r2-b2/R d00-inst ≠ d08-inst (INST) d00-profile.txt
r2-b2/R d00-inst ≠ d11-inst (INST) d00-profile.txt
r2-b2/R d02-p1-depth ≠ d03-p1 (P1) d02-first-dive.txt
r2-b2/R d03-zw ≠ d07-zw-eng (ZW_ENG) d03-monthly.txt

For r2-b1 forward, both replay graphs agree with each other and match the primary's entities/profiles, reviews, scores and active facts; each also contains one additional already-invalidated typed leads fact. The original graph comparison remains False in the table; historical facts were not filtered to make it pass.

The graph projection compares entity names/classes/descriptions/profiles, facts including invalidation flags, and review outcomes after normalizing creation IDs; it is not a byte-identical database claim. All attempts, raw model replies, identity states, per-KB metrics/missing anchors, first-error notes, scripts, binary/source hashes and validation logs are retained in the local evidence bundle. Credentials, account/settings rows, private keys and model weights are excluded.

Remaining: unmatched asynchronous adjudication/evidence inputs prevent a complete whole-pipeline equivalence claim. The fixed-state tests found no directly attributable new named-entity regression, but this narrower result cannot exclude every downstream scheduling effect. The next diagnostic needs a transaction-consistent capture at the facts tool read and its task boundary, not just the later model-request snapshot. Keep draft while that boundary is made reproducible; do not expand this PR into extraction/alias/adjudicator lifecycle work or claim overall accuracy improvement. No further model configuration is needed from the user.

Refs #1136, #1202, #1220.

…azy fact counts

Signed-off-by: Virtue039 <virtue039@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant