A chunk is named by the section it holds, and a document that ends with a heading keeps its last chunk by Maya-Kid · Pull Request #1026 · deeplethe/utopia · GitHub
Skip to content

A chunk is named by the section it holds, and a document that ends with a heading keeps its last chunk - #1026

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:fix/chunk-named-by-its-own-section
Oct 1, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:fix/chunk-named-by-its-own-section

Conversation

@Maya-Kid

Copy link
Copy Markdown
Contributor

Closes #1025. It also fixes a worse fault found on the way: a document whose text ends with a heading loses the text of its last chunk.

Problem

The last chunk of a document that ends with a heading is dropped.

  • When the text runs out with headings still pending, chunk_segments meant to give them a chunk of their own ("总好过丢掉").
  • It did that by replacing the body being collected with those headings, without flushing it first.
  • The text in that body never reaches the chunks: up to a full chunk, and for a short document all of it.

On dev 798fb24, this text gives one chunk, # Report\n\n## Appendix, and both paragraphs are gone:

# Report

First paragraph of the report.

Second paragraph of the report.

## Appendix

A note that ends with an empty section loses its content the same way. For example, # 周报 / 本周完成了三项工作。/ ## 附件 becomes # 周报\n\n## 附件. Extraction never reads the dropped text and search never finds it.

This applies to any file or page whose text ends with a heading line. Once #1018 lands, it also applies to a Word file whose last paragraph is a heading. So this should land first, or together with #1018.

A chunk closed at a section boundary is named after the next section (#1025).

  • heading() pushes a new section onto path as soon as it reads it, and keeps it in pending until the next piece is placed.
  • When that piece does not fit, place() flushes the body collected so far.
  • flush() took the breadcrumb from path. By then path holds the next section's headings, and has already popped the body's own heading at the same level.
  • enter_segment flushes the same way when the provenance changes after a heading.

With a budget of 14, # Report / ## Part A / First. / ## Part B / Second. / ### B.1 / Third. gave these names:

Chunk holds Name on dev
First. Report › Part B
Second. Report › Part B › B.1
Third. Report › Part B › B.1

The chunk text was right, because prefix_now leaves pending headings out. So headings_at in time resolution was not affected. chunks.heading is not read anywhere yet.

Change

  • heading() records the path as it was before the first of a run of new headings (before_pending).
  • While headings are still pending, flush() names the chunk by that path, because the pending headings belong to the next chunk. Otherwise it uses path, as before.
  • At the end of the text, the body being collected is flushed before trailing headings become their own chunk.

Nothing else in the packing changes. Chunk text, offsets, prefixes and the number of chunks stay the same. The one exception: a document that ends with a heading gains the chunk it used to lose.

Verification

Run on macOS (arm64): Rust 1.98.1; the workspace suite against PostgreSQL 16 with pgvector 0.8.2, with Poppler 26.06.

New tests in chunker.rs. Each fails on dev:

Test This branch dev
a_chunk_is_named_by_the_section_it_holds (the three chunks above) Report › Part A · Report › Part B · Report › Part B › B.1 Report › Part B · … › B.1 · … › B.1
a_chunk_closed_by_a_change_of_provenance_is_named_by_its_own_section (stated text under "Lease", then "## Schedule", then an OCR page) the stated chunk is "Lease" "Lease › Schedule"
a_document_that_ends_with_a_heading_keeps_its_last_text (the example above) the paragraphs in a chunk named "Report", then # Report\n\n## Appendix over the heading's own offsets one chunk, paragraphs gone

Other checks:

  • The existing chunker tests pass unchanged. cargo test --locked -p utopia-ingest: 112 passed.
  • cargo fmt --all --check and cargo clippy --locked --workspace --all-targets -- -D warnings are clean.
  • UTOPIA_TEST_REQUIRE_DB=1 UTOPIA_TEST_REQUIRE_PDFTOTEXT=1 cargo test --locked --workspace passes: 1,323 passed, 0 failed, 6 ignored.

Documents already imported keep their chunks until they are reprocessed. A document that ended with a heading gets its missing text back then.

…th a heading keeps its last chunk

Signed-off-by: dada-yan <BinjunYann@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks @Maya-Kid, and for catching the dropped last chunk on the way. Reviewed at 4008883: before_pending names a chunk closed at a section boundary after the section it holds, and the body is flushed before trailing headings become their own chunk. This lands before #1018, as you asked.

@WaylandYang
WaylandYang merged commit 71b3b8e into deeplethe:dev Oct 1, 2026
6 checks passed
@Maya-Kid
Maya-Kid deleted the fix/chunk-named-by-its-own-section branch October 1, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants