fix(muya): pair emphasis delimiters the way CommonMark does (#2086) by Jocs · Pull Request #5555 · marktext/marktext · GitHub
Skip to content

fix(muya): pair emphasis delimiters the way CommonMark does (#2086) - #5555

Merged
Jocs merged 7 commits into
developfrom
fix/nested-emphasis-delimiters-2086
Sep 26, 2026
Merged

Jocs merged 7 commits into
developfrom
fix/nested-emphasis-delimiters-2086

Conversation

@Jocs

@Jocs Jocs commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Closes #2086.

The bug

The live editor renders *a *b* c* as <em>a *b</em> c* and
**a **b** c** as <strong>a **b</strong> c**. CommonMark (and GitHub,
and pandoc) read both as nested emphasis:

$ echo '*a *b* c*' | pandoc --from commonmark
<p><em>a <em>b</em> c</em></p>

The export/static path is fine — it goes through marked, which passes the
whole emphasis section. It is muya's own inline lexer, which the editor
uses, that is wrong, so the two paths disagree on the same document.

Root cause

inlineRenderer/rules.ts matched emphasis with two regexps:

strong: /^(\*\*|__)(?=\S)([\s\S]*?[^\s\\])(\\*)\1(?!(\*|_))/,
em:     /^(\*|_)(?=\S)([\s\S]*?[^\s*\\])(\\*)\1(?!\1)/,

A lazy quantifier stops at the first syntactically plausible closer. But
CommonMark §6.2 says a closer binds to the nearest still-unmatched
opener
. In *a *b* c* the middle * is preceded by a space, so it can
only open — it claims the third *, and the outer pair is left to span
the whole line. No single pattern can express that, because the answer
depends on the runs in between.

The fix

New packages/muya/src/inlineRenderer/emphasis.ts implements the spec
algorithm for one inline string:

  1. Walk the string collecting * / _ delimiter runs, recording each
    run's length and whether it can open and/or close, from the code
    points on either side (left/right-flanking, plus the intraword _
    rule).
  2. Skip anything that binds more tightly than emphasis — code spans,
    autolinks, raw HTML tags, math spans, links, images, and backslash
    escapes — so their contents hold no delimiters. That is what makes
    *a `*`* one em around a code span rather than two stray asterisks.
  3. Pair closers with the nearest eligible opener, applying the rule of
    three and spending two characters at a time when both sides have them.
    Openers spend their characters right-to-left and closers left-to-right,
    so nested pairs nest properly and leftover characters stay literal
    (**foo* → *<em>foo</em>).

tryStrongEm now looks the current offset up in that map instead of
running a regexp. Token shape is unchanged (raw, marker, range,
children, backlash), so the renderer, format() and the marker
highlighting need no changes.

The CJK flanking widening from #4307 is carried over verbatim — it stays
additive — and lowerPriority still enforces §6.4 rule 17 on the
accepted span.

Result

The inline lexer against the CommonMark 0.31 "Emphasis and strong
emphasis" section: 82/132 → 131/132.

Newly correct includes all of the issue's cases plus, e.g.:

input before after
*a *b* c* <em>a *b</em> c* <em>a <em>b</em> c</em>
_foo _bar_ baz_ <em>foo _bar</em> baz_ <em>foo <em>bar</em> baz</em>
foo***bar***baz literal foo<em><strong>bar</strong></em>baz
**foo* <em>*foo</em> *<em>foo</em>
*foo**** literal <em>foo</em>***
*a `*`* *a <code>*</code>* <em>a <code>*</code></em>
foo ***** foo <strong>*</strong> foo ***** (literal)

The one example still failing is #354 (*£*bravo.): CommonMark 0.30
widened "Unicode punctuation" to the S (symbol) categories and
PUNCTUATION_REG still carries the 0.29 list. That is a separate gap
from delimiter pairing — the ASCII third of that same example,
*$*alpha., passes — and it is locked in the new suite so it can only be
removed by fixing it.

Tests

  • src/inlineRenderer/__tests__/emphasisDelimiterRuns.spec.ts — replays
    the whole emphasis section through the inline lexer with the same
    expected-failures contract test/spec/ uses: a listed example that
    starts passing fails the suite, so compliance can only go up. Plus
    direct assertions for the issue's two lines and the outer token's
    raw / range / children.
  • e2e/tests/inline/nested-emphasis-2086.spec.ts — real Chromium: both
    loading and typing *a *b* c* produce a nested <em> and round-trip
    back to the same markdown.

Verified locally: muya unit suite and test:spec unchanged apart from
the new tests; the muya Playwright suite shows the same failure set
before and after the change.

Follow-up commit (code review)

A review of the first commit found four things worth fixing, all now in
be4d6f4:

A run spent from both ends handed out the same character twice. A run
that closes one pair and then opens the next spends characters from
opposite ends, so a single "how many are left" counter is not enough.
*foo***bar* placed the second span one character early, and
**foo***bar** produced two overlapping spans — which cost the
tokenizer the whole <em>bar</em>. The run now tracks its unspent window
as [left, right): the closer advances left, the opener retreats
right.

backlash matches \(, so the backslash-math branch was dead. The
scan tried the escape rule before the math rules — the reverse of
INLINE_HANDLERS — so with texMathSingleBackslash on, the * inside
\(x*y\) stayed a delimiter and *a \(x*y\) b* lost its emphasis.
Reordered.

Each nesting level re-scanned its own content. That judged the
content's edge runs against the string boundary rather than the markers
now wrapped around them, so *_b.__* grew an inner <em> the enclosing
scan had deliberately refused. Spans are now keyed by absolute position
and the same map is handed down to the children.

lowerPriority per level made deep nesting quadratic. On
'*'*2000 + a + '*'*2000 — 1000 nested spans — a single tokenize took
4.6 s (34 ms before this PR). It is redundant now that the scan steps over
what binds more tightly, so it is gone from tryStrongEm and that set is
completed instead: reference links/images whose label resolves, GFM
extended autolinks (extent-trimmed, as tryAutoLinkExtension trims it),
and the tighter-binding check tryLink applies. Same input now tokenizes
in 4.3 ms. Patterns are matched with sticky clones rather than
substring(i), so the scan allocates nothing per position.

~~ is deliberately not in that set: GFM strikethrough is a delimiter
run in its own right, so *a ~~b*c~~ d* is <em>a ~~b</em>c~~ d*, as
cmark-gfm has it.

Diffing every line of the CommonMark corpus against the first commit
gives three changes, all towards the spec — CM 0.31 #523
*foo [bar* baz] now yields <em>foo [bar</em> baz].

Added a seeded property test for the invariant the spec examples do not
pin down and the first commit violated: no two spans may partially
overlap, and a span's two markers must match.

Note on the one red e2e run above: it was a flake. Running the full muya
Playwright suite locally with CI's settings gives the identical failure
set on both commits, and the re-run of that job on the same commit is
green.

🤖 Generated with Claude Code

Jocs and others added 7 commits September 25, 2026 10:57
The live editor's inline lexer matched `*` / `_` runs with a regexp, so
every opener took the first plausible closer to its right. `*a *b* c*`
came out as `<em>a *b</em> c*`, and `**a **b** c**` likewise — while the
static/export path (marked) rendered both as nested emphasis.

Replace the `strong` / `em` patterns with the spec's delimiter-run
algorithm: scan the inline string's runs, decide for each whether it can
open and/or close from its flanking characters, then let every closer
bind to the nearest still-unmatched opener, honouring the rule of three
and spending two characters at a time for strong. Runs inside a
construct that binds more tightly — a code span, autolink, raw HTML tag,
math span, link or image — are skipped, so `*a `*`*` is one em around a
code span.

The CJK flanking widening from #4307 is carried over unchanged, and
`lowerPriority` still guards §6.4 rule 17.

Against the CommonMark 0.31 "Emphasis and strong emphasis" section the
inline lexer goes from 82/132 to 131/132. The one remaining example is
#354, which needs `PUNCTUATION_REG` extended to CommonMark 0.30's
Unicode symbol categories — a separate gap, locked in the new suite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ring

Follow-up to the emphasis rewrite, from code review.

A run that closes one pair and then opens the next spends characters from
opposite ends, so a single "how many are left" counter is not enough: it
handed the same character to both pairings. `*foo***bar*` emitted the
second span one character early, and `**foo***bar**` emitted two
overlapping spans, which cost the tokenizer the whole `<em>bar</em>`.
Track the run's unspent window as `[left, right)` instead — the closer
advances `left`, the opener retreats `right`.

Three more problems, all of them the scan not agreeing with the
tokenizer:

  - `backlash` matches `\(`, so trying it before the math rules made the
    `case '\\'` branch unreachable and left the `*` inside `\(x*y\)` a
    delimiter. Order the scan like `INLINE_HANDLERS` does.
  - Each nesting level re-scanned its own content, which judged the
    content's edge runs against the string boundary rather than the
    markers now around them: `*_b.__*` grew an inner `<em>` the enclosing
    scan had deliberately refused. Key the spans by absolute position and
    hand the same map down to the children instead.
  - `lowerPriority` ran per level over the whole span, turning a
    keystroke on deeply nested emphasis into seconds of work
    (`'*'*2000 + a + '*'*2000`: 34 ms before the rewrite, 4.6 s after).
    It is redundant now that the scan steps over what binds more tightly,
    so drop it and complete that set instead: reference links and images
    whose label resolves, extended autolinks, and the tighter-binding
    check `tryLink` applies. The same input now tokenizes in 4.3 ms.

`~~` is deliberately not in that set: GFM strikethrough is a delimiter
run of its own, so `*a ~~b*c~~ d*` is `<em>a ~~b</em>c~~ d*`, as
cmark-gfm has it.

Over every line of the CommonMark corpus this changes three results
against the previous commit, all of them towards the spec — CM 0.31 #523
`*foo [bar* baz]` now yields `<em>foo [bar</em> baz]`.

Adds a seeded property test for the invariant the spec examples do not
pin down: no two spans may partially overlap, and a span's markers must
match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The delimiter scan has to know where a link, image, reference, autolink or
math span starts and ends, so it can step over the `*` and `_` inside one.
It answered that with its own copy of each handler's acceptance gate, and
two of the copies were already wrong:

  - a reference link was taken as inert without `tryReferenceLink`'s
    `lowerPriority` check, so a code span overrunning the brackets hid
    delimiters the tokenizer would go on to treat as real;
  - an extended autolink was taken as inert everywhere, though
    `tryAutoLinkExtension` only ever makes one at the top level — inside a
    `~~…~~` the scan skipped a bare URL the tokenizer renders as plain
    text, losing the emphasis in `~~www.x.com/a*b*c~~`.

Lift the three gates into `utils.ts` — `matchBracketed`, `matchReference`,
`matchExtendedAutoLink`, over a shared sticky `matchAt` so they can answer
at an index as well as at the head of a string — and call them from both
sides. `tryImage`, `tryLink`, `tryReferenceLink`, `tryReferenceImage` and
`tryAutoLinkExtension` shrink to one call each. `BACKSLASH_MATH_RULES`
moves to `rules.ts` for the same reason.
`collectRuns` walks the whole inline string, and three of its per-character
tests were regexps where a comparison does. On a 2000-character prose
paragraph the scan spent most of its time deciding that ordinary letters
start nothing:

  - the alphanumeric test before an extended autolink ran `.test()` on a
    freshly materialized one-character string, once per character;
  - an extended autolink was then attempted at every word start, 351 times,
    though its pattern needs `www.`, `http` or `@` somewhere;
  - the boundary test before it was a second regexp on another one-char
    string.

Character codes for the first and third, and one `includes` pass per scan
for the second. The scan drops from 0.047 ms to 0.005 ms, ~6% off a
keystroke's tokenize — this runs on the focused paragraph on every key.

Also names the two things the pairing loop kept spelling out: `unspent(run)`
for the `[left, right)` window the `IDelimiterRun` comment describes, and
`canPair` for the opener search, whose `break` used to mean "found" five
lines from a `break` meaning "not found".
…ated

Three places still describe the world before `emphasis.ts`:

  - `INLINE_HANDLERS` claims its array order "IS the rule-precedence
    contract". True for every handler but `tryStrongEm`, which sits third
    and defers to constructs handled further down the list.
  - `validateRules` was the "binds tighter than emphasis" set; emphasis no
    longer consults it and its only remaining caller is the emoji guard.
    Renamed `emojiValidateRules`, with a note that its membership is
    inherited from that era and is wider than an emoji guard needs.
  - `tokenizerFac`'s `emphasisSpans` parameter is defaulted, so the four
    recursions that must NOT inherit the parent's map say so only by
    omission — and they are correct for two opposite reasons. Written down
    at the parameter, because getting it wrong yields a mis-ranged token
    rather than a type error.
…ches

The suite's token-tree renderer carried a local `escapeHtml` that the
public `escapeHTML` already covers (the only difference, `'` → `&#39;`,
appears in none of the 132 emphasis examples), plus cases for `image`,
`html_escape` and `hard_line_break` that no input in the file can reach.
Removes the 154 comment lines this branch added across the emphasis
scanner, the lexer, the rules and the two new specs, keeping only the
functional directives (eslint-disable, @vitest-environment,
@ts-expect-error) and the pre-existing comments that merely moved file or
had to be reworded because the symbol they named is gone.
@Jocs
Jocs merged commit 18c724f into develop Sep 26, 2026
7 checks passed
@Jocs
Jocs deleted the fix/nested-emphasis-delimiters-2086 branch September 26, 2026 09:45
csells added a commit to csells/MarkText that referenced this pull request Oct 13, 2026
…arktext#5555)

CommonMark's rule of three reads the lengths of the delimiter runs that
hold the opener and the closer, and its openers-bottom classes read the
closer run's length. cmark and upstream MarkText (marktext#5555) use each run's
original length. Core used what earlier pairings had left of the run, so a
run that closed one pair and then opened the next was judged by its
remainder:
- `*foo***bar*` read as `<em>foo</em>**bar*`, and now reads as
  `<em>foo</em>*<em>bar</em>`, as upstream does.
- `**foo***bar**` read as `<strong>foo</strong>*bar**`, and now reads as
  `<strong>foo</strong><em>bar</em>*`.
- `..**a*   a*b` read as `..<em><em>a</em>   a</em>b`, and now reads as
  `..*<em>a</em>   a*b`.

Each inline delimiter now keeps its run's original length. The pairing
check and the openers-bottom and closer-index keys use it, while the run's
remaining length still decides how many characters a pairing spends.

Upstream's lexer also turned `a*\*b` into an empty emphasis. That is not
copied: CommonMark, the previous upstream and micromark keep the escaped
run literal, and a test pins it.

emphasis-original-run-length.spec.ts holds upstream's own cases from
emphasisDelimiterRuns.spec.ts, the corpus case above, the escaped run, and
every CommonMark 0.31.2 emphasis example. The first three cases fail
without this change.

E2E: emphasis-delimiter-runs.spec.ts types the closing asterisk of
`*foo***bar` with the real keyboard in the built app, sees two emphases and
saves `*foo***bar*`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support nested emphasis delimiters

1 participant