Containment recall reads the base's name facts through the predicate index instead of probing every entity's facts by gedaiu · Pull Request #1331 · deeplethe/utopia · GitHub
Skip to content

Containment recall reads the base's name facts through the predicate index instead of probing every entity's facts - #1331

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
re-cinq:perf/containment-from-name-facts
Oct 21, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
re-cinq:perf/containment-from-name-facts

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Part of #1322.

Thank you for the plan on the issue and for #1318, which made this a small change. containment_reviews now starts from the base's live known_as facts and joins the entity by id, instead of walking every live entity and probing its facts. The predicate is looked up as a scalar subquery, so it is an index condition on facts_live_predicate_idx, the same shape as name_vectors::PENDING_SQL. A canonical name is also a known_as fact, so one pass over the name facts covers both the canonical names and the aliases, and the correlated EXISTS is gone. Each entity is returned once however many of its names match. The filters, the escaping from #1320, the ordering, the limit and the output columns are unchanged, so the code that reads the rows is untouched.

I measured it on a seeded base with 39,254 live entities, 42,542 live known_as facts and 942,542 live facts. The machine is a Lenovo laptop (Intel Core Ultra 7 258V, 8 cores, 32 GB RAM, NVMe SSD) running Fedora 43, with Postgres 16 in Docker (pgvector/pgvector:pg16, default settings), on dev at f2e2a47. The numbers are the median of three warm EXPLAIN (ANALYZE, BUFFERS) runs. The laptop was doing other work at the time, so expect a few milliseconds of noise.

mention today name facts
Sherlock Holmes Museum (Latin, 7 matches) 206 ms 38 ms
星尘项目二期 (CJK, 3 matches) 200 ms 43 ms
src/auth/session_token.ts (code path, 14 matches) 215 ms 38 ms

Most of the time on today's query is JIT. Its planner cost is high enough that Postgres compiles it on every call. The new query stays far below the JIT thresholds. The plan has an InitPlan for the predicate, one index scan on facts_live_predicate_idx and no SubPlan. Shared buffers drop from about 83k to about 43k.

There is one difference, and I would like to check it with you. Entities created from a description have no known_as fact, and their canonical_name is the description text. Today's query pairs them by containment through that text, and the new one does not. ADR 0044 says a description is not a bridge for recall, and resolve_by_name already skips them with e.description IS NULL, so this brings containment in line with it. I compared the full result sets of today's query and the new one on nine mentions, and every row that differs is an entity with a description. The new query returns nothing that today's does not. If you would rather keep them, the change is a small extra branch on the entity side, and I am happy to add it.

The new tests are in a_similar_name_is_proposed_not_merged.rs. One pins the alias case. Mr. Holmes is paired with Sherlock Holmes through its alias Holmes, which is the case the name-fact join has to keep. The other pins the one change. An entity created from a description is not paired, while an ordinary entity with the same text in its known_as fact still is. That test fails on dev and passes with the change. The existing containment tests pass unchanged. I did not add a plan test, because the shape is visible in the EXPLAIN above and plan tests on small bases were flaky before.

I did not run the identity bench, since it needs a server with chat and embedding models. Instead I ran cargo test -p utopia-store --test store (328 passed) and cargo test -p utopia-store --lib (88 passed, 1 ignored), along with cargo fmt --check and cargo clippy --all-targets -- -D warnings. I also compared the pair sets against today's query on the seeded base as described above. It would be good to have the identity bench run on your side before merging.

Direction 2 as a lookup will come in a second PR, after the #1104 boundary lands.

🤖 Generated with Claude Code

…index instead of probing every entity's facts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
@WaylandYang

WaylandYang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

@WaylandYang
WaylandYang merged commit efb326d into deeplethe:dev Oct 21, 2026
6 checks passed
gedaiu added a commit to re-cinq/utopia that referenced this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants