fix(core): honour an explicit passage ID of 0 and key the offset map by string by Lesereingrape · Pull Request #427 · StarTrail-org/LEANN · GitHub
Skip to content

fix(core): honour an explicit passage ID of 0 and key the offset map by string - #427

Merged
ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
Lesereingrape:fix/explicit-passage-id-accepted-and-stringified
Sep 29, 2026
Merged

ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
Lesereingrape:fix/explicit-passage-id-accepted-and-stringified

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

What does this PR do?

LeannBuilder.add_text() selected the passage ID with

passage_id = metadata.get("id") or self._generate_passage_id(text)

so a caller-supplied ID went through a truthiness test. Two ways that loses data, both
visible through the public search API:

  1. {"id": 0} is discarded. Integer dataset IDs (a row id, a dataframe index) start
    at 0, and 0 is falsy, so the passage got _generate_passage_id(text) instead — which
    is str(len(self.chunks)), i.e. the ID of the next passage. Two passages end up
    sharing one ID in .passages.jsonl, and only one of them is reachable.
  2. A non-string ID breaks the offset map. The raw value was used as the key of
    docs.leann.passages.idx, while build_index() advertises the same ID to the backend as
    a string in .ids.txt and writes the idmap with str(sid). At search time the backend
    hands back the string, PassageManager.get_passage() looks it up in the int-keyed map,
    raises KeyError, and the enrichment loop logs Passage not found! and drops the hit
    — top_k=2 silently returns 1 result.

The fix honours an explicit ID whenever it is neither None nor empty, and stores it as a
string, which is the contract #422 (build_index_from_arrays()) already established for the
precomputed-vector path. add_text() is the remaining path where a caller's ID could be
rewritten.

tests/test_explicit_passage_id.py builds through the real build_index() and checks both
halves: that .ids.txt and the offset map agree on the string form, and that a zero-ID
passage still comes back from LeannSearcher.search() with its own text.

Related Issues

No existing issue. I looked for one: this is the un-converted sibling of #422.

Verification

Local, Windows, Python 3.12.11, base 7c268aa (origin/main):

  • The new tests RED with only the production line reverted:
    AssertionError: assert {'1', 7} == {'1', '7'} (int key in the offset map) and
    AssertionError: assert ['1', '1'] == ['1', '0'] (both passages share ID 1), GREEN
    2 passed with the fix.

  • Adjacent build/passage-ID/metadata suites on the same tree: pytest tests/test_explicit_passage_id.py tests/test_build_from_arrays.py tests/test_passage_id_scheme.py tests/test_metadata_filtering.py
    → 49 passed.

  • Backend-level suites: pytest tests/test_basic.py tests/test_hybrid_search.py tests/test_incremental_build.py tests/test_hnsw_rebuild_fallback.py → 18 passed, 3
    failed
    , and all three failures are this host missing a backend, raised in the backend
    lookup at api.py:464 before any passage ID is built:
    test_basic.py::test_backend_basic[diskann], test_incremental_build.py::test_ivf_incremental_add_then_remove_searchable,
    test_incremental_build.py::test_ivf_multiple_incremental_no_duplicates → ValueError: Backend 'diskann'/'ivf' not found or not registered. (only the hnsw wheel is installed
    here). Every hnsw-parametrized case in those files passed.

  • pre-commit run --files packages/leann-core/src/leann/api.py tests/test_explicit_passage_id.py
    (pinned pre-commit-hooks v5.0.0 + ruff v0.12.7) → every hook Passed.

  • ty check packages/leann-core/src apps tests (ty==0.0.17, the CI type-check) → 8
    diagnostics, all pre-existing and in files this PR does not touch (e.g. fcntl.LOCK_UN on
    Windows).

  • I did not run the full pytest tests/ locally: several suites need the diskann/ivf
    backends built from source and an OPENAI_API_KEY. CI covers them and this change is
    confined to add_text().

  • Tests pass (uv run pytest) — for the four files above; see the note on the rest

  • Code formatted (ruff format and ruff check)

  • Pre-commit hooks pass (pre-commit run --all-files) — run over the two changed files

AI assistance was used for this change.

…by string

add_text() picked the passage ID with `metadata.get("id") or
_generate_passage_id(text)`, so an integer ID of 0 fell through to the
generated one and collided with the passage that already owned it, and a
non-string ID keyed `.passages.idx` as an int while the backend advertises
it through `.ids.txt` as a string. Both made a built passage unreachable at
search time, where the KeyError branch silently dropped the hit.
@ASuresh0524
ASuresh0524 merged commit 37c5ae3 into StarTrail-org:main Sep 29, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants