Add a lexer for AsciiDoc by Madin-H23 · Pull Request #3421 · pygments/pygments · GitHub
Skip to content

Add a lexer for AsciiDoc - #3421

Open
Madin-H23 wants to merge 1 commit into
pygments:masterfrom
Madin-H23:add/asciidoc-lexer
Open

Madin-H23 wants to merge 1 commit into
pygments:masterfrom
Madin-H23:add/asciidoc-lexer

Conversation

@Madin-H23

Copy link
Copy Markdown

What

Fixes #1752 — adds a lexer for AsciiDoc, which Pygments had no support for at all (there is no AsciiDoc entry in the lexer mapping).

The lexer was written against the official AsciiDoc syntax quick reference and covers the parts that show up in ordinary documents:

  • the document title and the section titles (=, ==, ===, ...)
  • the document header: the author line, the revision line and attribute entries (:name: value)
  • delimited blocks — ---- (listing), ==== (example), **** (sidebar), ____ (quote), .... (literal) and ++++ (passthrough) — together with the block attribute line in front of them ([source,python], [NOTE])
  • list items (*, ., -, nested levels) and checkbox items (* [x])
  • labelled list items (term:: description)
  • admonitions (NOTE:, TIP:, IMPORTANT:, WARNING:, CAUTION:)
  • links (link:target[text], https://example.org[text]) and images (image::target[Alt])
  • comments (// and the //// block form) and attribute references ({name})
  • inline formatting: *bold*, _italic*, `monospace` and #mark#

The token types follow the other markup lexers in Pygments: Generic.Heading for the document title, Generic.Subheading for section titles, String.Backtick for monospace, Generic.Strong / Generic.Emph / Generic.EmphStrong for the inline marks, Name.Attribute for attribute entries, Name.Tag for link and image targets, Keyword for admonition labels, list markers and block attributes.

The inline marks follow the AsciiDoc rule that they have to sit on a word boundary, so some_var_name and text*not bold*text are left alone.

Scope

Tables (|===), the CSV table block and the more exotic inline macros are deliberately left out of this first version so that it stays reviewable; they can follow once the core is accepted.

Tests

tests/examplefiles/asciidoc/sample.adoc (+ generated golden) covers a document header with author, revision and attribute entries, section titles, all the inline marks, links, images, attribute references, labelled list items, nested and checkbox list items, an admonition block, a one-line admonition, a source block, a sidebar, an example, a quote, a literal block and both comment forms.

Verified locally:

  • pytest -q --ignore=tests/contrast — 5335 passed, 16 skipped
  • pytest tests/examplefiles -k asciidoc — 1 passed
  • ruff check --ignore UP031 ., scripts/check_crlf.py, scripts/detect_missing_analyse_text.py --skip-no-aliases, scripts/check_sources.py, scripts/count_token_references.py — all clean
  • the token stream round-trips to the input exactly, with no Error tokens, also for CRLF files, an empty file, a document without a title, an unterminated block, a table block and a file with only punctuation

@pbx — I saw your 2021 comment about starting a fork for this. If that work is still alive, please say so and I will step aside; otherwise, feel free to review this one.

AI usage disclosure: the lexer and its tests were prepared with AI assistance; all test results above were produced locally and reviewed.

AsciiDoc documents had no lexer at all.  The new lexer marks the document
title and the section titles, the document header (author line, revision
line and attribute entries), delimited blocks, list items, labelled list
items, admonitions, links, images, comments and the common inline
formatting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Please provide a AsciiDoc lexer.

1 participant