{{ message }}
Repository navigation
fix(core): honour an explicit passage ID of 0 and key the offset map by string - #427
Merged
ASuresh0524 merged 1 commit intoSep 29, 2026
Conversation
…by string
add_text() picked the passage ID with `metadata.get("id") or
_generate_passage_id(text)`, so an integer ID of 0 fell through to the
generated one and collided with the passage that already owned it, and a
non-string ID keyed `.passages.idx` as an int while the backend advertises
it through `.ids.txt` as a string. Both made a built passage unreachable at
search time, where the KeyError branch silently dropped the hit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What does this PR do?
LeannBuilder.add_text()selected the passage ID withso a caller-supplied ID went through a truthiness test. Two ways that loses data, both
visible through the public search API:
{"id": 0}is discarded. Integer dataset IDs (a row id, a dataframe index) startat 0, and
0is falsy, so the passage got_generate_passage_id(text)instead — whichis
str(len(self.chunks)), i.e. the ID of the next passage. Two passages end upsharing one ID in
.passages.jsonl, and only one of them is reachable.docs.leann.passages.idx, whilebuild_index()advertises the same ID to the backend asa string in
.ids.txtand writes the idmap withstr(sid). At search time the backendhands back the string,
PassageManager.get_passage()looks it up in the int-keyed map,raises
KeyError, and the enrichment loop logsPassage not found!and drops the hit—
top_k=2silently returns 1 result.The fix honours an explicit ID whenever it is neither
Nonenor empty, and stores it as astring, which is the contract #422 (
build_index_from_arrays()) already established for theprecomputed-vector path.
add_text()is the remaining path where a caller's ID could berewritten.
tests/test_explicit_passage_id.pybuilds through the realbuild_index()and checks bothhalves: that
.ids.txtand the offset map agree on the string form, and that a zero-IDpassage still comes back from
LeannSearcher.search()with its own text.Related Issues
No existing issue. I looked for one: this is the un-converted sibling of #422.
Verification
Local, Windows, Python 3.12.11, base
7c268aa(origin/main):The new tests RED with only the production line reverted:
AssertionError: assert {'1', 7} == {'1', '7'}(int key in the offset map) andAssertionError: assert ['1', '1'] == ['1', '0'](both passages share ID1), GREEN2 passedwith the fix.Adjacent build/passage-ID/metadata suites on the same tree:
pytest tests/test_explicit_passage_id.py tests/test_build_from_arrays.py tests/test_passage_id_scheme.py tests/test_metadata_filtering.py→ 49 passed.
Backend-level suites:
pytest tests/test_basic.py tests/test_hybrid_search.py tests/test_incremental_build.py tests/test_hnsw_rebuild_fallback.py→ 18 passed, 3failed, and all three failures are this host missing a backend, raised in the backend
lookup at
api.py:464before any passage ID is built:test_basic.py::test_backend_basic[diskann],test_incremental_build.py::test_ivf_incremental_add_then_remove_searchable,test_incremental_build.py::test_ivf_multiple_incremental_no_duplicates→ValueError: Backend 'diskann'/'ivf' not found or not registered.(only thehnswwheel is installedhere). Every
hnsw-parametrized case in those files passed.pre-commit run --files packages/leann-core/src/leann/api.py tests/test_explicit_passage_id.py(pinned
pre-commit-hooks v5.0.0+ruff v0.12.7) → every hook Passed.ty check packages/leann-core/src apps tests(ty==0.0.17, the CI type-check) → 8diagnostics, all pre-existing and in files this PR does not touch (e.g.
fcntl.LOCK_UNonWindows).
I did not run the full
pytest tests/locally: several suites need the diskann/ivfbackends built from source and an
OPENAI_API_KEY. CI covers them and this change isconfined to
add_text().Tests pass (
uv run pytest) — for the four files above; see the note on the restCode formatted (
ruff formatandruff check)Pre-commit hooks pass (
pre-commit run --all-files) — run over the two changed filesAI assistance was used for this change.