Repository navigation
Two people who share a name end as one entity through the bare-name entity #1136
Description
What happens
The identity bench has two people named 张伟, a finance director and an engineer. They should end as two entities. In the 16 bases run for #1134 (8 on dev, 8 on the branch; deepseek-v4-flash with thinking off, bge-m3) one entity ended up holding facts of both in 10: 3 of 8 on dev, 7 of 8 on the branch. #1134 does not touch this path (no name involved has digits, and both sides made 12 merges between 张伟 entities), so the gap between the sides reads as variance, and the variance is large.
Where the two people meet
In all three dev bases the meeting point is the entity named plainly 张伟. The titled entities (财务部总监张伟, 工程部工程师张伟) stay correct.
So there are two ways in:
- At resolution. A mention written as just
张伟goes to an existing entity of that exact name with no review, and when two such entities exist the mention can land on the wrong one. Two of the three bases. - At adjudication. Two entities of the same bare name are merged because nothing contradicts, though nothing connects them either. One of the three.
Merges of a bare 张伟 into a titled one were each backed by a matching passage and look right.
What to find out
- How an exact-name mention chooses between two entities of the same name, and whether its passage is consulted.
- Whether "same bare name, nothing in common" should be kept for a person instead of merged.
- The noise floor: one base adjudicated twice, so a change can be told from variance.
Dumps of both sides are kept (ib_dev-16bases, ib_fix-16bases). Any change here gets the identity bench before and after.
Activity
virtue039 commentedon Oct 10, 2026
I'd like to take the investigation and first minimal fix for #1136.
I'll start by tracing and reproducing the two paths you identified: exact-name mention resolution and same-name adjudication. I'll compare approaches for using passage context and requiring positive identity evidence, keeping ambiguous cases separate when the evidence is insufficient.
Before choosing the implementation, I'll prepare a small reproducible case and a scoped proposal here. Any change will be checked on the identity bench before and after, including repeated baseline runs to distinguish the effect from model variance and checking that genuine same-entity matches still resolve.
Please let me know if someone has already started on this.
WaylandYang commentedon Oct 10, 2026
Thank you for offering, and for #1178 this morning. Yes, please take it; nobody has started.
Your plan is the right order. A few things you are likely to run into:
scripts/bench/identity.mjsis the bench; a cheap model is fine as long as both sides use the same one.Thanks again for taking this on.
WaylandYang commentedon Oct 10, 2026
A retest, for whoever measures this next. Two builds that differ only by #1134 (0e24b32 and b05d85f), the same model and settings as before, 16 new bases a side on 2026-10-10:
The day before, the same measurement gave 3 of 8 and 7 of 8. In the last eight bases a side the adjudicator was as busy as before (19 and 22 merges involving a 张伟, 29 and 29 pairs of them kept apart) and none of the merges joined the two people; there was no merge of a bare
张伟with a bare张伟on either side.So the difference between the two days is larger than any difference between the two builds, with nothing changed on our side between them. The model is reached through a router, and its behaviour on these pairs is not stable from one day to the next. A fix here should be measured against a baseline taken in the same session, several bases a side, and a single day's count should not be read as an effect.
virtue039 commentedon Oct 11, 2026
The mention-resolution change in #1192 has been revised to reuse one unresolved entity and bound the repeated-mention human queue, as requested in review. The separate automatic exact-name adjudication evidence gate is #1199, following #1193.
Each PR has its own completed same-session qwen3.7-flash-2026-07-15 + local BGE-M3 evaluation: two rounds and eight fresh KBs per side. Their descriptions report all per-KB F1, entities, pending human reviews, false merges/splits and missing anchors. #1192's final mean F1 declined (0.5583 → 0.5321) and mean human reviews increased (0.375 → 1.125), despite the bounded repeated-mention regression passing. The adjudication gate's 3 observed withheld true same-entity pairs are reported explicitly; its small experiment does not demonstrate an overall quality gain. Combined lifecycle behavior is checked deterministically, not claimed as a joint live-model result. Both are partial changes, leaving the remaining singleton and qualified-name behavior for separate work.
virtue039 commentedon Oct 11, 2026
I will start with the fixed, scripted case set requested in the reviews of #1192 and #1199, leaving both production changes on hold.
The test-only slice will run the real extraction/resolution and adjudication entry points against local scripted replies, with source-identity gold labels fixed independently of those replies. It will cover wrong resolution attachment and high-confidence wrong adjudication, plus correct same/different controls, in both source/seed orders. No real-model request or new verifier is needed.
I will preserve strict gold failure on current dev. If these measurement fixtures are to land before a fix, CI will explicitly use a small known-failure budget and publish every gold violation: controls and new/worse failures still fail CI, while a budget pass means only that the known baseline did not worsen. The strict command and its current failing cases will be in the PR, rather than ignoring tests or asserting the wrong identities as correct.
virtue039 commentedon Oct 11, 2026
The fixed scripted measurement is now in #1202, branched directly from
dev(25fa9f4), without either held production change.Four source-labelled cases × both source/seed orders exercise real extraction/resolution and adjudication. The resolver has two existing bare-name candidates and a uniquely higher profile score for the wrong holder. The adjudicator receives a high-confidence
samefor two different employees. Same-personsameand different-persondifferentare controls in an open graph with no declared unique relation. Source identity labels, not model replies or canonical names, define the gold grouping.Current
dev: 4/8 gold disagreements, allfalse_mergein the two failure paths, both orders; four controls pass. All cases make exactly one local scripted request and add no pending Human reviews. Strict mode runs all eight and exits 101 on those four errors. The default CI ceiling allows only those named existing errors, accepts improvements and rejects new errors, control failures, Human growth and incomplete/script/DB measurements. CI publishes the gold table and raw log; green means no regression against that ceiling, not correct identity resolution. The budget/strict split is proposed for review, not assumed agreed.Calibration changed only the true-merge control reply to
different: both orders reportedfalse_splitand the default check failed. That temporary mutation was restored before the final workspace run: 1498 passed / 0 failed / 6 existing ignored, with DB/PDF guards, fmt/clippy/build and frozen-lockfile web checks passing. No real-model calls or new gate/lifecycle are included. #1192 and #1199 remain held while the cases and evidence are discussed.WaylandYang commentedon Oct 11, 2026
Thank you, that is the right order, and thank you for taking the hold in good part.
One suggestion on how the known failures land. Instead of a failure budget, please keep a named list in the test itself: the cases expected to fail today. A listed case must fail and every other case must pass, so a fix removes names from the list and a new failure cannot hide behind the count. That needs no CI flag.
Scripted replies and fixed gold labels are exactly what we need. Thanks again.
virtue039 commentedon Oct 13, 2026
Implemented the named-list approach in #1202 (
8050a8c), together with the cases requested in the latest PR review.Each listed
(case, order)must fail and every other check must pass, with no budget, CI flag, extra workflow step or ignored test. Today's four names are the passage-over-profile resolver case and the scalar-clue-to-request adjudicator case, each in both seed orders. The four paired controls pass. The resolver oracle is an exact pre-existing candidate ID; adjudication checks what each actual HTTP Record A/B section carries, not the correctness of a scripted verdict.Calibration confirmed a small code change can satisfy all eight while preserving controls; the unchanged list then correctly fails on unexpected passes, and removing those names makes the aggregate pass. Never attaching and dropping entity fact values were separately rejected by the controls. These prototype edits were fully restored, so the final diff is tests only.
Local workspace 1515 passed / 0 failed / 6 existing ignored, DB/PDF guards and required checks pass. The two production PRs remain held; this update submits the corrected measurement for review, without another model evaluation or production gate.
CI run 38410602919 passed all six jobs; the guarded workspace step executes the named-failure test. The maintainer has approved the rework and independently verified the list, confirming every review point is met.