fix(markdown): recognize setext headings by jayhemnani9910 · Pull Request #611 · VectifyAI/PageIndex · GitHub
Skip to content

fix(markdown): recognize setext headings - #611

Open
jayhemnani9910 wants to merge 2 commits into
VectifyAI:mainfrom
jayhemnani9910:fix/md-setext-headings
Open

jayhemnani9910 wants to merge 2 commits into
VectifyAI:mainfrom
jayhemnani9910:fix/md-setext-headings

Conversation

@jayhemnani9910

Copy link
Copy Markdown

Fixes #609. Stacked on #610 (it reuses _front_matter_end).

Only ATX (#) and whole-line bold headings become nodes. A heading written as text underlined with === or --- is treated as body text: its section merges into the one before it, and the sections below don't nest under it.

This treats a ===/--- line under plain paragraph text as a level 1/2 heading, as CommonMark does, using the whole paragraph as the title. Lines that aren't paragraph text don't qualify: list items, quotes, table rows, indented code, fenced code and leading front matter. So a thematic break after a blank line, --- under a list item, and front matter delimiters stay as they were.

Checked against markdown-it-py (CommonMark) on 2,815 .md files from other repos: setext headings it finds and the parser missed went from 91 to 0, with no new headings that markdown-it doesn't see. The remaining differences are older (HTML blocks, indented code, and the fence handling that #555 addresses).

Acme SDK            before: [Install, Usage]
========            after:  [Acme SDK > [Install, Usage]]

Tests: SetextHeadingTest (underlines, multi-line paragraph, four non-heading dash cases).

pytest tests   main: 814 passed   branch (with #610): 821 passed, 145 skipped

This touches extract_nodes_from_markdown, as #555 does, so whichever lands second needs a small rebase.

extract_node_text_content gives each node the text from its heading to
the next one, so anything above the first heading belonged to no node
and was dropped from the tree. A README's intro, a document whose title
is not an ATX heading, and a file with no headings at all (zero nodes)
lost that text from the index.

Keep it as a Preface node, as both PDF paths already do, at the level
of the first heading so the tree shape is unchanged. A leading YAML
front matter block is not part of it.
Only ATX (`#`) and whole-line bold headings became nodes. A title or
section written as text underlined with `===` or `---` was not a node,
so its text merged into the previous section, or into the Preface when
it came first, and the sections below it did not nest under it.

Treat a `===`/`---` line under plain paragraph text as a level 1/2
heading, as CommonMark does; the heading is the whole paragraph. List
items, quotes, table rows, indented code, fenced code and a leading
YAML front matter block are not paragraph text, so their dashes stay
what they were.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Markdown indexing drops text before the first heading, and setext headings

1 participant