{{ message }}
Repository navigation
A chunk is named by the section it holds, and a document that ends with a heading keeps its last chunk - #1026
Merged
WaylandYang merged 2 commits intoOct 1, 2026
Conversation
…th a heading keeps its last chunk Signed-off-by: dada-yan <BinjunYann@gmail.com>
WaylandYang
approved these changes
Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes #1025. It also fixes a worse fault found on the way: a document whose text ends with a heading loses the text of its last chunk.
Problem
The last chunk of a document that ends with a heading is dropped.
chunk_segmentsmeant to give them a chunk of their own ("总好过丢掉").On dev 798fb24, this text gives one chunk,
# Report\n\n## Appendix, and both paragraphs are gone:A note that ends with an empty section loses its content the same way. For example,
# 周报 / 本周完成了三项工作。/ ## 附件becomes# 周报\n\n## 附件. Extraction never reads the dropped text and search never finds it.This applies to any file or page whose text ends with a heading line. Once #1018 lands, it also applies to a Word file whose last paragraph is a heading. So this should land first, or together with #1018.
A chunk closed at a section boundary is named after the next section (#1025).
heading()pushes a new section ontopathas soon as it reads it, and keeps it inpendinguntil the next piece is placed.place()flushes the body collected so far.flush()took the breadcrumb frompath. By thenpathholds the next section's headings, and has already popped the body's own heading at the same level.enter_segmentflushes the same way when the provenance changes after a heading.With a budget of 14,
# Report / ## Part A / First. / ## Part B / Second. / ### B.1 / Third.gave these names:The chunk text was right, because
prefix_nowleaves pending headings out. Soheadings_atin time resolution was not affected.chunks.headingis not read anywhere yet.Change
heading()records the path as it was before the first of a run of new headings (before_pending).flush()names the chunk by that path, because the pending headings belong to the next chunk. Otherwise it usespath, as before.Nothing else in the packing changes. Chunk text, offsets, prefixes and the number of chunks stay the same. The one exception: a document that ends with a heading gains the chunk it used to lose.
Verification
Run on macOS (arm64): Rust 1.98.1; the workspace suite against PostgreSQL 16 with pgvector 0.8.2, with Poppler 26.06.
New tests in
chunker.rs. Each fails on dev:a_chunk_is_named_by_the_section_it_holds(the three chunks above)a_chunk_closed_by_a_change_of_provenance_is_named_by_its_own_section(stated text under "Lease", then "## Schedule", then an OCR page)a_document_that_ends_with_a_heading_keeps_its_last_text(the example above)# Report\n\n## Appendixover the heading's own offsetsOther checks:
cargo test --locked -p utopia-ingest: 112 passed.cargo fmt --all --checkandcargo clippy --locked --workspace --all-targets -- -D warningsare clean.UTOPIA_TEST_REQUIRE_DB=1 UTOPIA_TEST_REQUIRE_PDFTOTEXT=1 cargo test --locked --workspacepasses: 1,323 passed, 0 failed, 6 ignored.Documents already imported keep their chunks until they are reprocessed. A document that ended with a heading gets its missing text back then.