Search answers a service with each hit's score, ranks and source, and pages by gedaiu · Pull Request #1273 · deeplethe/utopia · GitHub
Skip to content

Search answers a service with each hit's score, ranks and source, and pages - #1273

Draft
gedaiu wants to merge 1 commit into
deeplethe:devfrom
gedaiu:feat/search-returns-its-scores
Draft

gedaiu wants to merge 1 commit into
deeplethe:devfrom
gedaiu:feat/search-returns-its-scores

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Part of #1271. Related: #1268 (an api push keeps its external_id, which the results now carry), #1262 (the graph reads a token gets).

What happened

POST /kbs/{id}/search took a session only, returned chunks without scores or ranks, had no paging, and fell back to full-text silently when the vector arm could not run. A backend could neither call it nor judge what came back.

What changed

  • utopia_search::rrf_fuse_scored returns each id with its fused score and its rank in every list (same k = 60, same tie-break); rrf_fuse wraps it.
  • retrieval::hybrid_scored(.., SearchOpts { top_k, offset, as_of }, embeddings) returns SearchPage { results, has_more, vector_used }; each result adds score, bm25_rank, vector_rank (1-based, null when the arm did not find it), source_id and external_id (for api and statements sources). Each arm takes max(offset + top_k + 1, 24) candidates, the extra one to tell has_more. hybrid wraps it with offset 0, so chat and tools are unchanged.
  • documents::chunks_by_ids joins the source and returns SourcedChunk (ChunkView flattened plus the two fields) instead of a second copy of the query.
  • The route takes SessionOrToken with require_reader_kb (made pub(crate) in place, as in A token reads the entities of the bases it reaches #1263), accepts offset, keeps top_k at 50 or less, and refuses offset + top_k > 100 with 422 page_too_deep. The response is {results, offset, top_k, has_more, vector_used}; every field it had is still there.
  • The SessionOrToken list in auth.rs, access-and-audit.md and 0014 decision 6 name chunk search; chat-and-mcp.md describes the scored response and that an RRF score ranks within one query only. No ADR owns hybrid search, so no new record.

Filters by source or filename come in the next part of #1271.

Tests

api::search_routes_tests, against a real database with five indexed chunks and no embedding model: a read token gets each hit with score 1/61, ranks, source and external_id, and vector_used: false; pages of two cover the five hits without overlap and has_more turns false on the last; a page past the hundredth hit is a 422. With the route opened to tokens but without scoring and paging, each fails for its own reason. rrf_fuse_scored has a unit test of the exact scores and ranks.

Run in rust:1-slim-bookworm against Postgres with UTOPIA_TEST_REQUIRE_DB=1: cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings clean; cargo test -p utopia-search 7 passed; cargo test -p utopia-server -p utopia-store: utopia-server 665 passed, 2 ignored, utopia-store store suite 311 passed.

…anks and source, and pages, so a backend can weigh and walk the results without a person's session

Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
@gedaiu

gedaiu commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant