mdast-util-from-markdown

mdast utility that turns markdown into a syntax tree

Contents

• What is this?
• When should I use this?
• Install
• Use
• API
  • fromMarkdown(value[, encoding][, options])
  • compiler([options])
  • preprocess(events)
  • takeExtension(combined, extension[, field])
  • creators
  • handles
• Types
  • Buffer
  • CharacterReferenceType
  • CompileContext
  • CompileData
  • Compiler
  • Config
  • Encoding
  • Enter
  • Event
  • Exit
  • Extension
  • Fragment
  • Handle
  • Handles
  • Node
  • OnEnterError
  • OnExitError
  • Options
  • ParseOptions
  • Point
  • Resume
  • StackedToken
  • StartPoint
  • TokenType
  • Token
  • TokenizeContext
  • Transform
  • Value
• List of extensions
• Syntax
• Syntax tree
• Security
• Related
• Contribute

What is this?

This package is a utility that takes markdown input and turns it into a markdown abstract syntax tree.

This utility uses micromark, which turns markdown into tokens, and then turns those tokens into nodes.

When should I use this?

If you want to handle syntax trees manually, use this.
When you just want to turn markdown into HTML, use micromark instead.
For an easier time processing content, use the remark ecosystem instead.

Install

This package is ESM only.

In Node.js (version 18+) with yarn:

yarn add @flex-development/mdast-util-from-markdown

See Git - Protocols | Yarn  for details regarding installing from Git.

In Deno with esm.sh:

import { fromMarkdown } from 'https://esm.sh/@flex-development/mdast-util-from-markdown'

In browsers with esm.sh:

<script type="module">
  import { fromMarkdown } from 'https://esm.sh/@flex-development/mdast-util-from-markdown'
</script>

Use

Say we have the following markdown file example.md:

## Hello, *World*!

…and our module example.mjs looks as follows:

import { fromMarkdown } from '@flex-development/mdast-util-from-markdown'
import { inspect } from '@flex-development/unist-util-inspect'
import { read } from 'to-vfile'

const file = await read('example.md')
const tree = fromMarkdown(String(file))

console.log(inspect(tree))

…now running node example.mjs yields:

root[1] (1:1-2:1, 0-19)
└─0 heading[3] (1:1-1:19, 0-18)
    │ depth: 2
    ├─0 text "Hello, " (1:4-1:11, 3-10)
    ├─1 emphasis[1] (1:11-1:18, 10-17)
    │   └─0 text "World" (1:12-1:17, 11-16)
    └─2 text "!" (1:18-1:19, 17-18)

API

This package exports the identifiers fromMarkdown, compiler, preprocess, and takeExtension.
The default export is fromMarkdown.

Additional exports are located at @flex-development/mdast-util-from-markdown/creators and @flex-development/mdast-util-from-markdown/handles.

fromMarkdown(value[, encoding][, options])

Turn markdown into a syntax tree.

Overloads

• (value: Value | null | undefined, encoding?: Encoding | null | undefined, options?: Options) => Root
• (value: Value | null | undefined, options?: Options | null | undefined) => Root

Parameters

• value (Value | null | undefined) — the markdown to parse
• encoding (Encoding | null | undefined, optional) — the character encoding for when value is Uint8Array
  • default: 'utf8'
• options (Options | null | undefined, optional) — the configuration options

Returns

(Root) mdast

compiler([options])

Create an mdast compiler.

👉 The compiler only understands complete buffering, not streaming.

Parameters

• options (Options | null | undefined, optional) — the configuration options

Returns

(Compiler) The mdast compiler

preprocess(events)

Prepare events before compilation begins.

Parameters

• this (mark.Context) — the event compilation context
• events (mark.Event[]) — the current list of events

Returns

(undefined) Nothing

takeExtension(combined, extension[, field])

Merge extension into the combined configuration.

Parameters

• combined (mark.Config) — the current configuration
• extension (mark.Extension) — the extension to merge
• field (keyof mark.Extension | undefined, optional) — the current extension field

Returns

(undefined) Nothing

creators

@flex-development/mdast-util-markdown/creators exports default Node factories by name.

handles

@flex-development/mdast-util-markdown/handles exports default token Handles by name.

Types

This package is fully typed with TypeScript.

Buffer

Start capturing output into a temporary node (type).

type Buffer = (this: CompileContext) => undefined

Parameters

• this (CompileContext) — the event compilation context

Returns

(undefined) Nothing

CharacterReferenceType

Union of character reference types (type).

type CharacterReferenceType =
  | 'characterReferenceMarkerHexadecimal'
  | 'characterReferenceMarkerNumeric'

CompileContext

The event compilation context (interface).

Event compilation consumes the Events of a parser to produce a syntax tree.

During compilation, event handlers orchestrate Node creation, enter nodes onto the stack, and exit them later.

The compilation config defines how TokenTypes map to compilation behavior. After the initial compilation, Transforms can be applied to manipulate the final tree.

Properties

• buffer (Buffer) — start capturing output into a temporary node.
  captured values can later be serialized with resume
• config (Config) — the compilation configuration
• data (CompileData) — info passed around; key/value store
• enter (Enter) — enter a node
• exit (Exit) — exit a node
• resume (Resume) — stop buffering and serialize the captured node
• sliceSerialize (TokenizeContext['sliceSerialize']) — get the string value of a token
• stack (Node[]) — the node stack
• tokenStack (StackedToken[]) — the token stack

CompileData

Interface of tracked data (interface).

interface CompileData {/* see code */}

When developing extensions that use more data, augment CompileData to register custom fields:

declare module 'mdast-util-from-markdown' {
  interface CompileData {
    mathFlowInside?: boolean | undefined
  }
}

To sync native augmentations, import @flex-development/mdast-util-from-markdown/mdast:

import type {} from '@flex-development/mdast-util-from-markdown/mdast'

Compiler

Turn micromark events into a syntax tree (type).

type Compiler = (this: void, events?: Event[] | null | undefined) => Root

Parameters

• events (Event[] | null | undefined, optional) — the list of events

Returns

(Root) mdast

Config

The event compilation configuration (interface).

Extends

• Extension

Properties

• afterEnter (Handle[]) — the list of listeners to call after a token is entered
• afterExit (Handle[]) — the list of listeners to call after a token is exiteded
• beforeEnter (Handle[]) — the list of listeners to call before a token is entered
• beforeExit (Handle[]) — the list of listeners to call before a token is exiteded
• canContainEols (string[]) — the list of token types where line endings are used
• enter (Handles) — record, where each key is a TokenType and each value is the handler to call when entering a token
• exit (Handles) — record, where each key is a TokenType and each value is the handler to call when exiting a token
• transforms (Transform[]) — the list of tree transforms to apply

Encoding

Encodings supported by TextEncoder (type).

See micromark-util-types for more info.

type Encoding =
  | 'utf-8' // always supported in node
  | 'utf-16le' // always supported in node
  | 'utf-16be' // not supported when ICU is disabled
  | (string & {}) // everything else (depends on browser, or full ICU data)

Enter

Enter a node (type).

type Enter = (
  this: CompileContext,
  node: Node,
  token: Token,
  onError?: OnEnterError | null | undefined
) => undefined

Parameters

• this (CompileContext) — the event compilation context
• node (Node) — the node to enter
• token (Token) — the token associated with node
• onError (OnEnterError | null | undefined, optional) — handle the case where another token is open, but closed by something else

Returns

(undefined) Nothing

Event

The start or end of a token amongst other events (type).

See micromark-util-types for more info.

type Event = ['enter' | 'exit', Token, TokenizeContext]

Exit

Exit a node (type).

type Exit = (
  this: CompileContext,
  token: Token,
  onError?: OnExitError | null | undefined
) => undefined

Parameters

• this (CompileContext) — the event compilation context
• token (Token) — the token associated with the node
• onError (OnExitError | null | undefined, optional) — handle the case where another token is open

Returns

(undefined) Nothing

Extension

A compiler extension (interface).

Compiler extensions can be used to change how tokens are turned into nodes, as well as apply transformations to the final syntax tree.

Properties

• afterEnter (Handle | Handle[] | null | undefined, optional) — the listener, or listeners, to call after a token is entered
• afterExit (Handle | Handle[] | null | undefined, optional) — the listener, or listeners, to call after a token is exiteded
• beforeEnter (Handle | Handle[] | null | undefined, optional) — the listener, or listeners, to call before a token is entered
• beforeExit (Handle | Handle[] | null | undefined, optional) — the listener, or listeners, to call before a token is exiteded
• canContainEols (string[] | null | undefined, optional) — the list of token types where line endings are used
• enter (Handles | null | undefined, optional) — record, where each key is a TokenType and each value is the handler to call when entering a token
• exit (Handles | null | undefined, optional) — record, where each key is a TokenType and each value is the handler to call when exiting a token
• transforms (Transform[] | null | undefined, optional) — the list of tree transforms to apply

Fragment

A temporary node (interface).

Properties

• children (mdast.PhrasingContent[]) — the list of children
• type ('fragment') — the node type

Handle

Handle a token (type).

type Handle = (this: CompileContext, token: Token) => undefined | void

Parameters

• this (CompileContext) — the event compilation context
• token (Token) — the token to handle

Returns

(undefined | void) Nothing

Handles

Record, where each key is a TokenType and each value is an event handler (type).

type Handles = Partial<Record<TokenType, Handle>>

Node

Union of nodes that can be created during event compilation (type).

👉 Note: Also exported as StackedNode.

type Node = Fragment | Root | RootContent

OnEnterError

Handle the case where the right token is open, but is closed by the left token, or because end of content was reached (type).

type OnEnterError = (
  this: Omit<CompileContext, 'sliceSerialize'>,
  left: Token | undefined,
  right: Token
) => undefined

Parameters

• this (Omit<CompileContext, 'sliceSerialize'>) — the event compilation context
• left (Token | undefined) — the exiting token
• right (Token) — the open token

Returns

(undefined) Nothing

OnExitError

Handle the case where the right token is open, but is closed by exiting the left token (type).

type OnExitError = (
  this: Omit<CompileContext, 'sliceSerialize'>,
  left: Token,
  right: Token
) => undefined

Parameters

• this (Omit<CompileContext, 'sliceSerialize'>) — the event compilation context
• left (Token) — the exiting token
• right (Token) — the open token

Returns

(undefined) Nothing

Options

Configuration options (interface).

Extends

• ParseOptions

Properties

• extensions (micromark.Extension[] | null | undefined, optional) — extensions for micromark to change how tokens are produced
• from (StartPoint | null | undefined, optional) — the point before the first character in the content.
  node positions will be relative to this point
  • default: { column: 1, line: 1, offset: 0 }
• mdastExtensions ((Extension | Extension[])[] | null | undefined, optional) — extensions for this utility to change how tokens are turned into nodes

ParseOptions

Options for parsing markdown (interface).

See micromark-util-types for more info.

Point

A location in the source document and chunk (interface).

See micromark-util-types for more info.

Resume

Stop capturing output and serialize the captured node (type).

type Resume = (this: CompileContext) => string

Parameters

• this (CompileContext) — the event compilation context

Returns

(string) The serialized node

Root

A document fragment or a whole document (interface).

See mdast for more info.

StackedToken

List containing an open token on the stack, and an optional error handler to use if the token isn't closed properly (type).

👉 Note: Also exported as TokenTuple.

type StackedToken = [token: Token, handler: OnEnterError | undefined]

StartPoint

The point before the first character in the content (type).

type StartPoint = Omit<Point, '_bufferIndex' | '_index'>

Token

A span of chunks (interface).

See micromark-util-types for more info.

TokenizeContext

A context object that helps with tokenizing markdown constructs (interface).

See micromark-util-types for more info.

TokenType

Union of registered token types (type).

See micromark-util-types for more info.

Transform

Transform the syntax tree (type).

👉 Note: Returning null, undefined, or void will preserve tree.

type Transform = (this: void, tree: Root) => Root | null | undefined | void

Parameters

• Root (Root) — the tree to transform

Returns

(Root | null | undefined | void) The serialized node

Value

The contents of a file.

See micromark-util-types for more info.

type Value = Uint8Array | string

List of extensions

• mdast-util-directive — directives
• mdast-util-frontmatter — frontmatter (YAML, TOML, more)
• mdast-util-gfm — GFM
• mdast-util-gfm-autolink-literal — GFM autolink literals
• mdast-util-gfm-footnote — GFM footnotes
• mdast-util-gfm-strikethrough — GFM strikethrough
• mdast-util-gfm-table — GFM tables
• mdast-util-gfm-task-list-item — GFM task list items
• syntax-tree/mdast-util-math — math
• syntax-tree/mdast-util-mdx — MDX
• syntax-tree/mdast-util-mdx-expression — MDX expressions
• syntax-tree/mdast-util-mdx-jsx — MDX JSX
• syntax-tree/mdast-util-mdxjs-esm — MDX ESM

Syntax

Markdown is parsed according to CommonMark. Extensions can add support for other syntax. If you’re interested in extending markdown, more information is available in micromark’s readme.

Syntax tree

The syntax tree is mdast.

Security

As markdown is sometimes used for HTML, and improper use of HTML can open you up to a cross-site scripting (XSS) attack, use of mdast-util-from-markdown can also be unsafe.

When going to HTML, use this utility in combination with hast-util-sanitize to make the tree safe.

Related

• mdast-util-to-markdown — serialize mdast as markdown
• micromark — parse markdown
• remark — process markdown

Contribute

See CONTRIBUTING.md.