Why

Fixes #1557. With reranking enabled, a vector or hybrid request whose page reached past the fixed reranked prefix ran retrieval twice: once for the expanded window, and again at the fixed window's size to rebuild the prefix (a second query embedding and vector query, plus a second FTS pass for hybrid). This replaces closed #1560 on top of the scope-parameterized SearchReader from the #1558 stack, so the fix covers the project route and the scoped QUERY /v2/search/ route.

What changed

• CandidateWindow records each chunk's position in the adapter ranking. ranked_rows(position_bound) returns the rows, scores, and passages a window of that bound would produce. A larger window only appends chunks, so this reproduces the fixed window from the request's own window.
• SemanticSearch._prefix_rows reads the fixed prefix at _rerank_candidate_limit chunks. With structured filters, the fixed window widens until it admits enough rows, so the bound extends to cover the leading admitted rows. Passages past the bound stay out of the prefix.
• Hybrid builds its prefix by fusing fts_results[:K] with the vector prefix rows. Fusion is now a pure fuse() used by both the full fusion and the prefix. Equal fused scores are ordered by key.
• Hybrid tail rows are ordered by where each row first entered either ranking: its FTS rank, or its best chunk's vector position counted before chunks collapse into rows. Without this, a vector row that appears late could move ahead of tail rows an earlier page already returned.
• _rerank_and_paginate now takes the prefix and the tail from its caller instead of stable_rows. vector_only and hybrid lose the candidate_limit, apply_rerank, and emit_observability_log parameters, which only existed for the recursive refetch.
• SQLite FTS breaks score ties with search_index.id ASC, as Postgres already did. A smaller FTS window is then always a prefix of a larger one.
• stable_pool_refetched is removed from the internal search trace. It was not exposed in the inspect schema.

Salvaged from #1560

• Kept: the pagination corpus fixture and the main real-backend test. It pages at sizes 1, 4, and 13 and compares against one large page, checking identical ids, identical scores, identical reranker document batches, and span counts per request. It is now parametrized over the project route and a scoped SearchReader. Also kept: the hybrid collapse tail-order test and the fixed-prefix-passages test, both ported to SemanticSearch._run_vector_query. The call-count assertions updated from [18, 8]/[90, 80] to [18]/[90], including the test-int/semantic ones.
• Dropped: the production code, which targeted the pre-Support database-wide search with optional project filtering #1558 repository methods. Also dropped: the sqlite-vec candidate_rank/fixed k=4096 adapter change and its tests (dropped adapter slots, pending-manifest slots, unordered-stream threshold). This design counts positions in the hydrated stream, so it does not need adapter ranks. A fixed k=4096 cost about 6x on shallow sqlite-vec queries in a local microbenchmark (1.8 ms to 10.5 ms at 10k vectors).
• Dropped: tests/semantic_search_helpers.py and the ty extra-paths change. The new test imports helpers from tests.repository.test_rerank_pipeline, like other suites do.

Known limit

sqlite-vec picks arbitrarily among exactly equal distances at its k cutoff. Windows of different sizes can therefore disagree on which identical vectors they hold. This already affects every sqlite-vec window boundary on main (tail pages, non-reranked pagination). This change extends it to the prefix boundary in the rare case where exact-duplicate passages sit there. The test corpus gives each passage its own distance. If we want tie determinism, the option is the fixed-k query from #1560, scoped to reranked requests. Postgres passes the same suite with tied vectors.

Testing

• The new regression tests fail on main: 2 embeddings per deep reranked request, and a misordered collapse tail. They pass on this branch.
• just fast-check: passed.
• SQLite: tests/repository tests/api/v2 tests/services tests/mcp: 3600 passed. test-int (not semantic/benchmark): 607 passed.
• Postgres (BASIC_MEMORY_TEST_POSTGRES=1): stable-pagination, rerank pipeline, trace, fusion, reader, vector pagination/filter/threshold, pgvector index, and search router suites: 174 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BF4TkgAJpVrgwzrhemkdig